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Abstract 

System  acquisition  decision  makers  are  frequently  charged  with  choosing  a 
single  system  from  a  set  of  feasible  possibilities  that  could  best  fulfill  the  needs  of  their 
organizations.  While  numerous  rules  and  regulations  are  already  in  place  for  both 
commercial  and  government  acquisitions  to  ensure  the  acquisitions  are  conducted 
fairly,  decision  makers  need  greater  support  than  rules  and  regulations  alone  can 
provide.  The  acquisition  decision  is  a  complex  data  analysis  problem,  where  the 
decision  maker  must  analyze  multiple  candidate  systems  on  a  number  of  performance 
and  cost  metrics.  To  understand  this  multivariate  environment,  decision  makers  must 
analyze  the  system  data  at  multiple  levels  of  reasoning.  This  research  proposes  a 
decision  support  tool  that  best  supports  system  acquisition  decision  makers  by 
providing  them  with  graphical  representations  displaying  how  well  candidate  systems 
fulfill  their  organizations'  needs. 

System  acquisition  decisions  require  support  of  three  basic  levels  of  reasoning 
(Data  Processing,  Information  Aggregation,  and  Knowledge  Synthesis)  in  order  to 
perform  system  trade-offs  on  relevant  system  metrics.  To  test  how  well  decision 
support  tools  could  support  system  acquisition  decision  makers,  two  graphical  decision 
support  tools  were  designed:  a  traditional  separable  display  and  a  new  configural 
display  named  Fan  Visualization  (FanVis).  To  compare  the  effectiveness  of  FanVis 
against  a  traditional  separable  display,  an  experiment  was  conducted  where 
participants  answered  a  series  of  system  acquisition  questions  across  the  three  levels  of 
reasoning. 

Analysis  of  the  experimental  results  indicate  that  FanVis  and  the  separable 
displays  support  a  system  acquisition  decision  maker,  but  to  different  degrees  across 
the  three  levels  of  reasoning.  Comparatively,  participants  tended  to  have  higher 
performance  on  Knowledge  Synthesis  tasks  using  FanVis,  while  they  tended  to  have  a 
higher  performance  on  Data  Processing  tasks  using  the  separable  display.  When 
examining  subjective  measures,  FanVis  was  the  preferred  tool  of  choice.  Through  use  of 
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an  eye  tracking  device,  it  was  further  determined  that  participants  also  exhibited  erratic 
fixation  patterns  on  those  questions  that  were  answered  incorrectly  compared  to  those 
answered  correctly.  Further,  it  was  determined  that  FanVis  allowed  participants  to 
maintain  more  efficient  gaze  patterns  regardless  of  task,  whereas  participants  used  less 
efficient  gaze  patterns  in  the  separable  display  for  some  tasks.  Additionally, 
participants  tended  to  spend  a  greater  frequency  of  time  fixating  on  relevant  elements 
in  FanVis  while  completing  Knowledge  Synthesis  tasks,  while  the  opposite  was  true  for 
Data  Processing  tasks,  suggesting  that  performance  and  time  spent  fixating  on  relevant 
information  is  correlated.  From  the  results  of  this  experiment,  a  set  of  design 
implications  was  created  for  future  system  acquisition  decision  support  tools. 

Thesis  Supervisor:  M.L.  Cummings 

Title:  Associate  Professor  of  Aeronautics  and  Astronautics 
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1.  Introduction 


1.1.  Motivation 

Each  year,  billions  of  dollars  of  revenue  are  generated  by  acquisition  and 
procurement  processes,  herein  referred  to  as  acquisitions.  Acquisitions  allow  systems  to 
be  appropriated  by  an  organization  through  a  contractual  agreement  with  a  supplying 
organization.  To  determine  which  system  best  fits  the  needs  of  their  organizations, 
decision  makers  must  compare  numerous  candidate  systems  in  a  decision  process 
known  as  source  selection.  The  source  selection  milestone  is  a  critical  step  to  complete, 
yet  is  often  the  most  difficult  portion  of  an  acquisition,  since  it  requires  decision  makers 
to  objectively  understand  large-scale  system  trade-offs  through  the  analysis  of  a 
complex  multivariate  set  of  quantitative  data. 

In  an  attempt  to  ensure  that  acquisitions  are  completed  properly,  the  federal 
government  has  created  numerous  rules  and  regulations  that  are  followed  by  all  federal 
executive  agencies.  In  the  commercial  world,  individual  organizations  have  instantiated 
their  own  standards,  rules  or  regulations.  Frequently,  however,  in  both  the  federal  and 
commercial  worlds,  acquisition  processes  are  conducted  improperly.  This  mistake  often 
results  in  the  acquisition  of  a  substandard  or  costly  system  and  could  result  in  the 
organization's  failure  to  accomplish  its  end  goal  [1]. 

Various  initiatives  and  studies  have  been  implemented  to  improve  the 
acquisition  process  including  new  acquisition  processes  within  the  construction 
industry  [2,  3],  the  software  industry  [4-6],  as  well  as  advanced  processes  for  any  type  of 
acquisition  [7,  8].  However,  these  new  acquisition  processes  focus  primarily  on  creating 
methods  and  algorithms  for  data  management  with  little  regard  for  how  best  to  display 
system  acquisition  data  trade-offs  to  decision  makers. 


15 


The  quantitative  data  to  be  analyzed  in  a  system  acquisition  decision  consists  of 
performance  and  cost  metrics  of  the  various  candidate  systems.  Candidate  systems 
meet  these  metrics  to  various  degrees,  typically  represented  to  the  decision  makers  as 
individual  data  points.  Thus,  the  resultant  data  set  for  how  well  candidate  systems  meet 
an  organization's  needs  is  inherently  multivariate  in  nature  and  can  be  quite  large 
depending  on  the  complexity  of  the  system.  Analyzing  this  large,  multivariate  set  of 
data  is  complex,  underscoring  the  importance  of  providing  decision  makers  with  an 
intuitive  depiction  of  this  information  in  an  understandable  format. 

Additionally,  while  analyzing  each  of  these  data  points  individually  is  objective, 
the  overall  analysis  of  the  data  set  can  become  subjective  as  trade-offs  must  be  made 
between  the  various  desired  metrics.  Decision  makers  will  make  the  best  system 
acquisition  decisions  when  they  understand  the  information  being  presented  to  the 
greatest  degree  possible.  While  there  are  a  handful  of  displays  that  have  been 
developed  to  provide  acquisition  decision  support,  these  displays  have  typically  been 
developed  in-house  with  no  documented  demonstration  that  they  provide  the 
necessary  information  to  improve  decision-making.  The  lack  of  support  for  a  decision 
maker,  who  is  faced  with  a  complex,  multivariate  decision,  presents  a  significant 
research  gap.  To  this  end,  this  research  focuses  on  displaying  system  acquisition 
information  in  a  more  intuitive,  principled  format. 

This  work  proposes  a  new  graphical  display  that  supports  ecological  perception, 
that  of  presenting  data  in  such  a  manner  so  that  users  directly  perceive  relationships 
within  the  data  [9].  Displaying  acquisition  data  in  a  graphical  manner  was  chosen 
because  graphical  formats,  in  general,  have  been  shown  to  be  more  helpful  for 
understanding  quantitative  data  than  conventional  statistical  computations  and  data 
tables  [10,  11].  Further,  a  configural  display  that  supports  ecological  perception  is 
conjectured  to  improve  system  acquisition  decisions  as  compared  to  traditional 
spreadsheet-based  bar  graphs  and  line  charts.  This  thesis  describes  the  reasoning 


16 


behind  this  hypothesis,  the  design  of  a  resultant  configural  display,  and  the  experiment 
used  to  validate  the  configural  tool's  increased  effectiveness  in  system  acquisition 
decisions  against  a  more  traditional  spreadsheet  decision  support  tool. 

1.2.  Problem  Statement 

To  complete  system  acquisition  decisions,  decision  makers  must  understand  how 
potential  systems  fulfill  their  organizations'  needs.  Through  the  years,  an  increase  in  the 
number  of  systems  along  with  an  enhanced  range  of  system  functionalities  has  caused 
system  acquisition  to  become  progressively  more  difficult.  Decision  makers  must  search 
through  larger  trade  space  sets,  determining  the  similarities  and  differences  among 
candidate  systems  in  an  attempt  to  choose  the  best  system  to  meet  the  desired  criteria. 
This  choice  requires  that  the  decision  maker  utilize  multiple  levels  of  reasoning,  ranging 
from  simple  data  comparisons  to  complex  knowledge  synthesis.  This  thesis  seeks  to 
determine  the  type  of  decision  support  system  that  best  aids  decision  makers  as  they 
utilize  the  multiple  levels  of  reasoning  required  to  successfully  complete  a  system 
acquisition  decision. 

1.3.  Research  Objectives 

To  address  this  goal,  the  following  research  objectives  were  posed: 

•  Objective  1:  Determine  the  motivating  principles  for  a  system  acquisition 
decision  support  tool.  In  order  to  achieve  this  objective,  current  acquisition 
practices  and  standards  were  researched,  as  described  in  Chapter  2.  In  addition, 
current  data  analysis  displays  were  analyzed  including  how  they  have  been  used 
in  previous  applications. 

•  Objective  2:  Develop  a  system  acquisition  decision  support  tool.  From  the 
motivating  principles  described  in  Objective  1,  a  system  acquisition  decision 
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support  tool  was  designed,  described  in  Chapter  3.  Included  is  a  discussion  of 
the  design  principles  applicable  to  this  display. 

•  Objective  3:  Evaluate  the  effectiveness  of  the  system  acquisition  decision 
support  tool.  To  address  this  objective,  human  participant  experimentation 
(Chapters  4  and  5)  was  conducted  to  analyze  how  well  the  system  acquisition 
decision  support  tool  was  able  to  support  an  acquisition  decision  compared  to  a 
traditional,  separable  decision  support  tool. 

1.4.  Thesis  Organization 

This  thesis  is  organized  into  the  following  chapters: 

•  Chapter  1,  Introduction,  describes  the  motivation  and  research  objectives  of  this 
thesis. 

•  Chapter  2,  Background,  outlines  the  scope  and  current  practices  within  system 
acquisitions.  This  chapter  identifies  current  gaps  within  the  system  acquisition 
process,  and  how  a  new  graphical  system  acquisition  decision  support  tool  can 
address  these  gaps. 

•  Chapter  3,  Display  Design,  provides  an  analysis  of  the  performance  of  current 
display  designs  on  tasks  similar  to  those  of  a  system  acquisition  decision.  The 
results  from  this  analysis  generate  a  set  of  criteria  that  provide  guidance  for  the 
design  of  a  configural  acquisition  decision  support  tool.  This  research  proposes 
this  tool  will  support  system  acquisitions  decision  maker  to  a  greater  degree  than 
a  separable  decision  support  tool,  which  is  also  described  in  this  chapter. 

•  Chapter  4,  Experimental  Evaluation,  describes  the  human-performance 
experiment  used  to  test  the  hypothesis  of  this  research.  Details  include  a 
discussion  of  participants,  procedures,  and  experimental  design. 

•  Chapter  5,  Results,  presents  the  results  of  the  human-performance  experiment  on 
such  metrics  of  accuracy,  speed,  subjective  appeal,  and  eye  fixation  patterns. 
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•  Chapter  6,  Conclusions  and  Future  Work,  compares  the  results  of  the  human- 
performance  experiment  with  the  research  hypothesis.  These  results  are 
described  on  the  basis  of  performance,  subjective  appeal  and  the  cognitive 
strategies  of  the  participants.  Based  upon  these  results,  a  set  of  design  and 
experimental  recommendations  are  given.  Finally,  future  work  necessary  to 
integrate  a  system  acquisition  decision  support  tool  into  current  practice  is 
described. 
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2.  Background 


When  conducting  an  acquisition,  decision  makers  within  a  soliciting 
organization  must  choose  which  system  could  best  fulfill  their  organizational  needs. 
Passing  this  milestone  is  often  referred  to  as  source  selection  and  is  accomplished  by 
comparing  proposed  candidate  systems  to  the  organization's  needs.  These  needs  often 
include  cost  criteria,  a  set  of  requirements  the  system  must  meet,  high-level  system 
characteristics  the  system  must  exhibit,  and  other  key  system  attributes.  Information 
regarding  the  candidate  systems  are  supplied  by  organizations  responding  to  the 
soliciting  organization's  Request  for  Proposal  (RFP). 

Source  selection  for  simpler  systems  may  be  fairly  straightforward,  as  decision 
makers  only  have  to  analyze  a  small  set  of  data  and  a  single  system  could  be  quickly 
pinpointed  as  the  best  system  to  acquire.  However,  for  sophisticated  systems,  decision 
makers  must  analyze  a  large,  complex,  multivariate  data  set  where  it  is  likely  no  one 
system  will  emerge  as  the  one  clear  winner.  Graphical  decision  support  tools  could 
greatly  benefit  decision  makers  in  these  situations  by  displaying  the  data  in  an  easily 
understandable  manner.  This  chapter  describes  the  current  practices  of  system 
acquisitions,  the  general  scope  of  such  decisions,  and  proposes  how  a  decision  support 
tool  could  best  fulfill  current  needs. 

2.1.  System  Acquisition  Practices 

Various  practices  have  been  established  by  both  the  federal  government  and 
individual  organizations  in  order  to  address  these  complicated  decisions.  Current 
system  acquisition  practices  in  both  commercial  industry  and  the  government  strive  to 
enable  a  system  acquisition  environment  that  focuses  on  obtaining  the  best  value  while 
maintaining  a  level  of  accountability,  integrity  and  a  degree  of  competition  [12-14],  This 
section  describes  the  various  standards,  rules,  regulations,  and  practices  that  have  been 
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established  and  how  the  proposed  decision  support  tool  can  further  enable  the  goals  of 
these  various  organizations.  Organizations  within  the  United  States  were  primarily 
studied,  but  similar  standards,  rules,  regulations  and  practices  have  been  established  by 
organizations  outside  of  the  country  as  well. 

2.1.1.  Federal  Government  Standards 

Federal  agencies  initiate  and  complete  acquisitions  through  the  use  of  government 
appropriated  funds.  One  of  the  largest  blocks  of  funds  is  allocated  toward  the  various 
branches  of  the  Department  of  Defense  through  the  National  Defense  Authorization 
Act.  In  Fiscal  Year  (FY)  2008,  the  authorized  funds  for  procurement  purposes  totaled 
$91.9  billion  and  were  distributed  across  the  various  DoD  branches,  as  illustrated  in 
Figure  1  [15].  In  an  effort  to  ensure  these  funds  are  spent  efficiently,  the  government  has 
established  a  number  of  regulations  to  help  federal  agencies  complete  the  acquisition 
process. 


Figure  1:  Department  of  Defense  2008  funding  in  billions 
The  Federal  Acquisition  Regulation  (FAR)  system  is  the  primary  set  of  regulations 
used  by  nearly  all  Federal  Executive  agencies  when  acquiring  supplies  and  services 
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with  government  appropriated  funds.  There  are  53  parts  to  the  FAR,  each  of  which 
consists  of  one  or  more  subsections  that  dictates  specific  regulations.  For  example, 
subsection  7.105  outlines  the  contents  a  written  acquisition  plan  should  include,  and 
subsection  13.106  outlines  regulations  for  soliciting  competition,  conducting  the 
evaluation  of  quotations  or  offers,  the  award  process,  and  the  required  documentation 
when  utilizing  simplified  acquisition  procedures  [12]. 

Through  each  of  these  subsections,  the  FAR  outlines  the  many  steps  required  to 
complete  a  system  acquisition.  To  begin  a  system  acquisition,  the  soliciting  organization 
should  first  develop  an  acquisition  plan  which  includes  information  pertaining  to  the 
acquisition  objectives,  the  required  capability  of  the  system,  design  trade-off,  budgeting, 
and  more.  In  all,  there  are  29  different  portions  of  the  acquisition  process  that  should  be 
considered  and  documented.  From  this  acquisition  plan,  the  soliciting  organization 
should  create  a  request  for  proposal  which  includes  the  criteria  they  will  evaluate 
candidate  systems  against.  The  exact  evaluation  criteria  are  dependent  upon  the 
system's  specifications  but  could  include  cost  or  price,  past  performance,  technical 
evaluation,  cost  information,  and  small  business  subcontracting.  When  candidate 
proposals  are  submitted,  the  decision  makers  within  the  soliciting  organization  must 
then  decide  which  candidate  system  best  meets  their  evaluation  criteria. 

However,  the  details  of  how  this  decision  should  be  completed  are  left  to  the 
discretion  of  the  decision  makers.  The  FAR  only  stipulates  that  the  decision  "shall  be 
based  on  a  comparative  assessment  of  proposals  against  all  source  selection  criteria"  in 
the  request  for  proposal  [12].  There  are  neither  specific  regulations  as  to  how  this 
evaluation  should  take  place  nor  a  commonly  agreed  upon  set  of  tools  the  system 
acquisition  decision  team  can  use  to  make  the  final  source  selection  decision.  Yet 
decision  makers  require  the  most  support  in  this  task,  as  they  are  analyzing  highly 
complex  multivariate  information  for  systems  that  are  typically  very  costly.  As  there  are 
currently  no  mandated  or  recommended  tools,  there  is  a  significant  gap  in  the  federal 
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acquisition  process.  A  graphical  system  acquisition  decision  support  tool  could  help 
bridge  this  gap,  allowing  decision  makers  to  make  the  best  decision  by  enabling  them  to 
understand  the  complex  information  in  the  most  objective  manner  possible. 

2.1.2.  Commercial  World  Standards 

Individual  organizations  outside  of  the  federal  government  must  also  obtain  goods 
and  services  from  various  organizations,  but  unlike  the  federal  government,  are  not 
subjected  to  the  same  rules  and  regulations.  However,  like  the  federal  government, 
these  organizations  must  also  ensure  acquisitions  are  completed  correctly,  efficiently, 
and  with  a  high  degree  of  accountability.  In  response,  organizations  have  created  their 
own  acquisition  strategies,  standards  and  guidelines  that  are  specifically  tailored  to 
their  unique  business  model. 

While  these  standards  are  extensive,  flexibility  within  them  allows  individuals  and 
organizations  to  establish  innovative  acquisition  strategies.  These  strategies 
demonstrate  new  processes  which  companies  can  use  to  improve  their  source  selection 
process.  A  variety  of  strategies  have  been  suggested,  ranging  from  simpler  strategies, 
which  add  a  new  criterion  to  the  selection  process  [3],  to  more  complex  methods,  which 
introduce  artificial  neural  networks  to  approximate  the  real  world  experience  of  an 
acquisition  manager  [8].  However,  these  strategies  lack  a  formal  analysis  of  how 
decision  makers  visualize  the  results  of  the  analysis.  Ultimately,  the  decision  maker 
must  be  able  to  understand  the  results  of  each  analysis,  including  why  one  system  may 
be  better  than  another.  While  the  manner  in  which  these  results  are  obtained  is  critical, 
poor  understanding  of  the  results  can  break  down  the  acquisition  process. 

2.2.  Scope  of  Acquisitions 

An  acquisition  is  typically  initiated  through  a  Request  for  Proposal  (RFP).  The 
soliciting  organization  creates  and  distributes  the  RFP  to  request  responses  from 
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multiple  supplying  organizations.  The  soliciting  organization  will  include  details  within 
the  RFP  regarding  the  metrics  that  each  system  must  meet  in  order  to  be  considered  a 
viable  option.  While  these  metrics  vary  from  project  to  project,  they  generally  include 
detailed  functional  requirements  and  high-level  system  characteristics  that  must  be 
satisfied  by  the  system.  Additional  metrics  could  be  included  depending  on  the  needs 
of  the  soliciting  organization. 

The  RFP  is  submitted  early  in  the  life  cycle  of  a  system  which  consists  of  multiple 
phases,  beginning  at  the  User  Requirements  Definition  phase,  and  ending  in  the 
Disposal  phase  [16],  as  shown  in  Figure  2  The  system  life  cycle  consists  of  many 
milestone  decisions,  two  of  which  are  displayed  as  diamonds  (Figure  2)  as  they  pertain 
to  the  focus  of  this  thesis.  In  order  to  advance  to  the  engineering,  manufacturing,  and 
development  phase,  the  soliciting  organization  must  choose  which  system  to  develop 
and  implement.  This  source  selection  decision  requires  that  the  soliciting  organization 
evaluate  the  various  proposals  received  based  upon  the  metrics  outlined  in  the  RFP.  It  is 
this  evaluation  process  which  could  most  benefit  from  the  use  of  an  advanced  graphical 
decision  support  tool.  This  section  describes  the  system  metrics  analyzed  in  the  context 
of  this  research,  and  the  resultant  levels  of  reasoning  that  decision  makers  within  a 
soliciting  organization  must  use  to  complete  this  decision. 
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Figure  2:  Generic  life  cycle  of  a  system  (adapted  from  [16]) 
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2.2.1.  System  Metrics 

Nearly  all  system  acquisitions  should  consider  at  least  the  following  three  critical 
system  metrics,  based  upon  systems  engineering  principles  described  in  the  subsequent 
paragraphs.  Other  system  metrics  such  as  risk,  delivery  schedule,  and  past  performance 
could  be  included  as  future  work. 

1)  The  degree  to  which  functional  requirements  are  met 

2)  The  degree  to  which  non-functional  requirements  ("-ilities")  are  met 

3)  System  Cost  by  Life  Cycle  Phase 

Functional  requirements  describe  the  actions  necessary  to  achieve  a  specific 
objective  [17].  For  example,  in  the  case  of  an  aircraft,  these  functional  requirements 
could  include:  ability  to  safely  takeoff  within  a  distance  of  2,000  feet,  establish  a  climb  of 
at  least  1,000  feet  per  minute,  cruise  at  or  above  200  knots,  and  land  within  a  distance  of 
1,000  feet.  The  functional  requirements  in  turn  produce  sub-functional  requirements 
such  as  configure  an  aircraft  to  take  off  conditions,  start  engine,  accelerate  to  necessary 
take  off  speed,  and  establish  a  positive  rate  of  climb.  From  these  sub-functional 
requirements,  lower  level  requirements  are  defined  until  the  necessary  pieces  of 
hardware  and  software  are  identified. 

The  "-ilities,"  are  characteristics  that  a  system  must  exhibit  such  as  reliability, 
adaptability,  sustainability,  modularity  and  usability  [17].  The  "-ilities"  can  play  as 
crucial  of  a  role  as  functional  requirements  when  choosing  a  system  [18],  as  they  define 
specific  behaviors  or  attributes  that  must  be  met  in  order  to  be  useful  to  the  end  user. 
The  system  must  also  be  economically  feasible,  since  the  soliciting  organization  will 
often  have  a  limited  set  of  funds  allocated  to  the  acquisition.  In  some  instances,  these 
funds  will  be  allocated  during  particular  portions  of  the  system's  life  cycle,  thus  it  is 
critical  that  the  decision  maker  understand  the  total  cost  and  the  cost  of  each  life  cycle 
phase  (Figure  2). 
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The  following  trade  space  variables  were  identified  to  convey  how  well  a  system 
meets  each  of  the  system  metrics  listed.  These  trade  space  variables  include: 

•  The  degree  to  which  functional  requirements  are  met  (f.r.  met) 

•  The  degree  to  which  "-ilities"  are  met  ("-ilities"  met) 

•  Total  cost 

•  Cost  per  sub-functional  requirement  (Cost  per  sub-f.r.) 

•  Life  cycle  cost 

By  including  the  system's  cost  allocations  in  addition  to  total  cost  and  life  cycle  cost, 
decision  makers  can  determine  the  cost-benefit  of  the  system  at  the  functional 
requirement  level.  In  addition,  decision  makers  can  also  perform  a  wide  variety  of 
tradeoffs,  such  as  determining  if  a  large  allocation  of  funds  was  appropriated  on  a 
functionality  that  ultimately  would  not  meet  its  requirement,  or  determine  if  a  single 
functional  or  sub-functional  requirement  was  driving  cost  unnecessarily. 

2.2.2.  Levels  of  Reasoning 

The  ultimate  goal  of  a  decision  maker  is  to  answer  feasibility  questions  (i.e.,  does  a 
system  meets  a  set  of  selection  criteria?),  or  optimality  questions  (i.e.,  in  the  case  of 
deciding  among  competitive  systems,  which  system  best  meets  the  desired  criteria?). 
The  selection  criteria  could  emphasize  cost  over  functional  requirements  met,  or  could 
emphasize  that  all  proposed  functional  requirements  be  met,  regardless  of  cost.  Due  to 
the  cost-benefit  nature  of  this  process,  a  decision  maker  needs  to  make  comparisons 
within  and  across  the  system  metrics  discussed  in  the  previous  section.  Frequently, 
decision  makers  will  use  their  subjective  opinions  to  complete  these  cost-benefits  trade¬ 
offs,  especially  in  the  acquisition  of  sophisticated  systems.  Thus,  a  decision  support 
display  should  provide  straightforward  and  intuitive  data  integration  to  support  these 
comparisons  in  the  most  objective  manner  possible.  In  essence,  there  are  three  general 
levels  of  reasoning  that  will  occur  for  these  complex  acquisition  decisions. 
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1)  Data  Processing:  Low-level  reasoning  that  compares  values  within  a  single 
constraint.  For  example,  determining  which  competitive  system  has  the 
overall  greatest  cost. 

2)  Information  Aggregation:  Mid-level  reasoning  that  integrates  of  data  across  a 
single  constraint.  For  example,  determining  which  system  meets  all  the 
functional  requirements,  as  multiple  values  of  similar  type  must  be  integrated 
before  a  conclusion  can  be  drawn. 

3)  Knowledge  Synthesis:  High-level  of  reasoning  that  requires  the  integration  of 
information  across  multiple  constraints.  For  example,  determining  which 
system  has  the  lowest  cost,  meets  all  "-ilities,"  and  meets  all  functional 
requirements. 

These  three  levels  of  reasoning  allow  a  decision  maker  to  formulate  answers  to 
different  questions.  Given  the  data  processing  example  above,  the  focus  may  strictly  be 
a  straightforward  cost  comparison  among  systems,  which  does  not  require  any  data 
integration.  However,  for  a  more  complex  problem  such  as  determining  the  best  system 
based  on  functional  requirement  analysis,  the  answer  cannot  easily  be  found  from 
simple  data  manipulation.  Thus,  any  systems  acquisition  decision  support  tool  must  be 
able  to  support  both  simple  data  manipulation  and  comparison,  as  well  as  higher  order 
data  operations. 

2.3.  Background  Summary 

Overall,  there  is  a  large  body  of  literature  that  supports  the  claim  that  a  decision 
support  tool  could  help  aid  an  acquisition  decision  for  complex  system.  System 
acquisitions  require  decision  makers  to  process  complex  multivariate  data  on  multiple 
levels  of  reasoning,  as  well  as  understand  all  candidate  system  information  and  perform 
cost-benefit  trade-offs  to  determine  which  system  could  best  fulfill  the  needs  of  their 
organizations.  While  the  processes  leading  up  to  this  decision  are  highly  structured. 
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there  is  an  overall  lack  of  guidance  regarding  to  how  the  decision  maker  can  best 
understand  which  candidate  system  meets  the  organization's  needs  in  the  most 
objective  manner  possible.  The  next  chapter.  Display  Design,  describes  how  a  graphical 
decision  support  tool  could  help  support  system  acquisition  decision  makers 
understand  these  complex  multivariate  problems. 
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3.  Display  Design 


The  primary  function  of  a  visual  display  device  is  to  impart  information  to  a  user, 
which  can  be  conveyed  in  different  ways  through  any  number  of  interfaces  such  as  a 
website,  a  text  document,  or  a  data  analysis  tool.  A  data  analysis  tool  could  simply  be  a 
spreadsheet  of  numbers  with  digital  or  analog  information,  or  a  graphical 
representation  of  those  numbers.  Previous  research  has  indicated  the  advantages  and 
disadvantages  of  each  display  type  on  a  variety  of  tasks.  This  chapter  describes  the 
results  of  this  previous  research  and  how  it  pertains  to  system  acquisition  decisions. 
Further,  this  chapter  describes  the  design  and  implementation  of  two  system 
acquisition  decision  support  tools:  one  is  a  traditional  separable  display  while  the  other 
is  a  new  configural  display. 

3.1.  Components  of  a  Data  Analysis  Display 

A  data  analysis  display  consists  of  elements  that  either  represent  data  or  tools  that  a 
user  can  manipulate  to  access  additional  data,  manipulate  the  data  or  manipulate  the 
representation  of  the  data.  Those  elements  that  represent  data  are  considered  to  be  more 
useful  to  the  end  user,  as  the  user  can  directly  abstract  the  information  he  or  she  is 
trying  to  acquire  [10].  Tools  to  manipulate  the  data  can  be  extremely  useful,  but  the 
majority  of  a  user's  time  should  be  spent  viewing  the  data  elements  rather  than  trying 
to  understand  the  analysis  tools.  The  data  elements  in  a  system  acquisition  decision 
support  tool  should  encode  the  trade  space  variable  data  as  defined  in  Section  2.2.1  for 
each  of  the  candidate  systems  being  analyzed,  including  the  degree  to  which  functional 
requirements  are  met,  the  degree  to  which  "-ilities"  are  met,  total  cost,  cost  per  sub¬ 
functional  requirement,  and  life  cycle  cost. 

It  is  the  selection  and  arrangement  of  the  data  elements  within  a  display  that  make  a 
display  useful  for  a  certain  set  of  tasks.  Therefore,  different  types  of  displays  were 
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analyzed  in  order  to  determine  how  to  design  the  data  elements  within  a  system 
acquisition  decision  support  tool  to  best  fit  the  needs  of  a  decision  maker  conducting  a 
system  acquisition  decision. 

3.2.  Previous  Display  Designs 

A  digital  (respectively,  analog)  display  is  a  non-graphical  display  that  simply  shows 
a  number  or  value  for  each  continuous  (respectively,  discrete)  variable  being  imparted 
to  the  user.  For  a  system  acquisition  decision,  these  variables  are  the  trade  space 
variables  described  in  2.2.1.  In  this  trade  space,  two  of  the  trade  space  variables  are 
discrete  (the  degrees  to  which  the  system  requirements  and  "-ilities"  are  met)  while 
three  are  continuous  (cost  variables).  Users  of  digital  and  analog  displays  have  been 
shown  to  have  poorer  performance  compared  to  users  of  graphical  displays  in  terms  of 
analyzing  complex  data  [19,  20],  though  the  contrary  has  been  shown  dependent  upon 
the  task  and  the  experience  of  the  user  [11],  However,  for  tasks  requiring  the  integration 
of  information,  graphical  displays  have  been  found  to  be  superior  [11],  These  types  of 
tasks  are  necessary  in  system  acquisition  decisions,  hence  graphical  displays  are  the 
focus  of  this  chapter. 

Most  graphical  data  analysis  displays  can  further  be  categorized  as  either 
separable  or  configural  displays.  Separable  displays,  such  as  bar  graphs  generated  from 
a  spreadsheet,  assign  unique  representations  to  each  state  variable  [21],  Configural 
displays  map  individual  variables  in  such  a  way  to  create  emergent  features  which 
allow  users  to  perceive  higher  level  interactions  or  constraints  among  individual  state 
variables  through  the  means  of  natural  mapping  [22],  These  interactions  are  created  by 
determining  which  relationships  exist  between  the  high  and  low-level  information,  and 
presenting  the  low-level  information  in  such  a  way  that  these  relationships  are  shown. 

For  example.  Figure  3  illustrates  how  a  set  of  system  requirements  (requirements 
A-E)  are  met  in  a  digital  display  format  (Figure  3a),  a  graphical  separable  display 
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(Figure  3b),  and  a  graphical  configural  display  (Figure  3c).  The  degree  to  which  the 
functional  requirement  is  being  met  is  displayed  on  a  five  point  Likert  scale  [23],  where 
1  signifies  "does  not  meet"  and  5  signifies  "greatly  exceeds".  In  this  example,  each 
display  allows  the  user  to  extract  the  functional  requirement  information,  but  the  user 
may  find  this  task  to  be  easier  with  one  of  the  displays  over  the  other. 
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Figure  3:  Examples  of  display  types  based  upon  functional  requirement  data 
Previous  research  has  been  conducted  to  determine  the  advantages  and 
disadvantages  of  separable  and  configural  displays  for  varying  degrees  of  reasoning 
processing  difficulty.  For  data  analysis  tasks,  it  has  been  found  that  configural  displays 
generally  improve  a  user's  performance  while  completing  integration  problems  [24,  25] 
while  separable  displays  result  in  improved  performance  while  completing  problems 
that  do  not  require  integration  [26].  It  is  believed  that  because  system  acquisition 
decisions  require  both  the  integration  and  comparison  of  information,  a  configural 
display  could  best  support  these  types  of  decisions.  This  hypothesis  will  be  described 
in  further  detail  later  in  this  chapter. 
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3.3.  Fan  Visualization 


Before  the  hypothesis  that  a  configural  decision  support  tool  could  support  a  system 
acquisition  decision  to  a  greater  degree  compared  to  a  more  traditional  separable 
decision  support  tool  could  be  tested,  a  configural  decision  tools  had  to  be  designed  and 
implemented.  Fan  Visualization,  or  FanVis  for  short,  is  a  system  acquisition  decision 
support  tool  which  consists  of  a  series  of  configural  displays  displaying  both  high  and 
low-level  information  by  incorporating  emergent  properties.  Basic  features  such  as 
color,  shape,  location,  and  size  were  integrated  into  the  design  display  to  promote 
preattentive  processing  [27].  Preattentive  processing  allows  the  user  to  quickly  observe 
and  extract  large  multi-element  displays  into  the  user's  preattentive  system  to  later  be 
joined  in  the  focused  attention  system  into  coherent  objects  [28]. 

In  total,  there  are  5  different  two-dimensional  views  in  FanVis:  1)  the  System  View, 
2)  the  Multi- System  View,  3)  the  Functional  Requirement  View,  4)  the  Comparison 
View,  and  5)  the  "-ility"  View.  All  the  views  were  built  upon  the  System  View  to 
provide  the  decision  maker  with  different  perspectives  of  the  acquisition  trade  space. 

3.3.1.  FanVis  Architecture 

The  views  within  the  decision  support  tool  are  supported  by  an  architecture 
programmed  in  Java.  Each  display  is  built  within  the  shell  shown  in  Figure  4.  This  shell 
consists  of  four  parts: 

1.  The  functional  buttons:  Allow  the  decision  maker  to  add,  or  delete 
components  of  each  system.  These  buttons  are  highlighted  by  the  green 
dotted  line. 

2.  Tree  structure:  Lists  all  trade  space  variables  within  the  system  acquisition 
trade  space  such  as  the  functional  requirements  and  "-ilities"  for  each 
candidate  system.  Also  where  the  decision  maker  can  change  any  of  these 
trade  space  variables.  In  addition,  the  tree  structure  allows  the  decision  maker 
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to  switch  which  component  of  the  trade  space  is  being  viewed.  This  structure 
is  highlighted  by  the  red  dash-dot  line. 

3.  Tabs:  Allow  the  decision  maker  to  select  the  view  to  be  displayed.  The  tabs 
are  highlighted  by  the  purple  dashed  line. 

4.  View  space  area:  Area  where  the  actual  view  will  be  displayed.  Currently  no 
view  is  displayed.  Decision  makers  can  change  between  the  views  through 
the  tab  structure.  The  decision  maker  can  also  expand  particular  views  to 
other  screens,  allowing  access  to  either  a  single  or  multiple  views. 


Figure  4:  The  overall  shell  of  FanVis 

3.3.2.  The  System  View 

The  main  structure  of  FanVis  is  similar  to  a  radar  chart  where  variables  (in  this  case, 
the  functional  requirements  of  the  proposed  system)  are  represented  by  axes  which 
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originate  from  a  central  point.  Each  system  in  the  design  space  is  represented  by  a 
polygon  in  the  System  View,  such  as  in  Figure  5. 

The  vertices  of  a  system's  polygonal  representation  intersect  the  functional 
requirement  axes  at  particular  points  along  those  axes  to  demonstrate  how  well  the 
system  meets  each  particular  functional  requirement.  The  axes  scales  are  a  five  point 
Likert  scale  [23]  with  the  following  delineations:  1)  Does  not  Meet  Requirements  (closest 
to  the  central  point),  2)  Partially  Meets  Requirements,  3)  Meets  Requirements  (middle 
point,  shown  in  red),  4)  Exceeds  Requirements,  and  5)  Greatly  Exceeds  Requirements 
(furthest  from  the  central  point).  Faint  lines  connect  the  axes  along  this  five  point  scale 
to  provide  a  visual  anchor. 


In  all  likelihood  the  decision  maker  would  want  the  system  to  be  on  or  outside  the 
red  line  (center  pentagon  in  Figure  5),  which  represents  that  the  functional 
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requirements  are  being  met  at  some  minimum  level.  Up  to  ten  functional  requirements 
can  be  viewed  at  a  given  time,  and  each  functional  requirement  can  have  up  to  fifteen 
sub-functional  requirements  due  to  space  limitations  of  the  visualization  for  a  21-inch 
desktop  computer  with  a  resolution  of  1280  x  1024  pixels,  16  bit  resolution. 

Each  vertex  contains  a  fan  comprised  of  individual  blades,  which  represent  the  sub¬ 
functional  requirements'  within  the  functional  requirement  on  that  axis.  These  blades 
are  shaded  according  to  how  expensive  the  sub-functional  requirement  is  in  relation  to 
the  most  expensive  sub-functional  requirement  for  that  system.  The  system's  most 
expensive  sub-functional  requirement  of  the  entire  system  will  thus  be  completely 
black,  while  the  least  expensive  will  be  primarily  white  (if  there  are  significant 
differences  in  the  costs).  This  expense  could  either  be  the  total  cost  or  one  of  the  life 
cycle  phase  costs  as  chosen  by  the  decision  maker  through  a  selection  menu  (Figure  6). 


Figure  6:  System  View  of  FanVis  displaying  the  selection  menu 
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Additionally,  the  selection  menu  in  Figure  6  allows  the  decision  maker  to  show  data 
labels  as  desired.  These  data  labels  provide  the  name  of  a  data  element  along  with  the 
data  value.  Decision  makers  are  able  toggle  these  data  labels  on  and  off  to  avoid 
cluttered  displays.  They  can  also  scroll  over  individual  data  elements  with  their  mouse 
to  obtain  this  information. 

In  addition  to  the  options  provided  by  the  selection  menu,  the  decision  maker  can 
easily  add,  delete,  or  modify  the  system  data  within  the  tool  through  tools  in  the  tree 
structure  and  functional  buttons  (Figure  7).  For  instance,  if  the  decision  makers  wanted 
to  input  data  regarding  a  new  functional  requirement,  they  would  select  the  system  in 
the  tree  structure  (Step  1)  then  click  "Add  Functional  Requirement"  in  the  functional 
buttons  (Step  2).  As  shown  in  Figure  7,  this  creates  a  new  functional  requirement.  In  a 
similar  manner  sub-functional  requirements  can  be  added  to  new  or  existing  functional 
requirement,  as  well  as  costs  or  new  systems. 


Figure  7:  Example  of  FanVis  interactivity  within  the  System  View 
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If  the  decision  maker  would  like  to  delete  the  functional  requirement  (or  any 
variable  within  the  system  trade  space),  he  or  she  only  has  to  click  on  the  requirement 
or  variable  in  the  tree  structure  then  click  "Delete"  in  the  functional  buttons.  Further, 
the  decision  maker  can  change  the  value  of  any  trade  space  data  point  by  double 
clicking  on  the  variable  in  the  true  structure  then  type  in  the  new  value. 

Analyzing  Figure  5,  there  are  several  emergent  features  in  this  view  that  are 
important  to  note,  the  first  of  which  is  the  polygon's  shape  and  size;  the  area  that  the 
system  encompasses  is  a  reflection  of  the  degree  of  success  with  which  the  functional 
requirements  are  met.  This  follows  a  natural  mapping  of  bigger  is  better  for  this 
objective.  Additionally,  if  a  system  is  balanced  in  terms  of  how  it  meets  the  functional 
requirements,  it  will  have  a  balanced  shape,  which  follows  a  user's  appeal  towards 
symmetry  [9].  For  example,  in  Figure  5  the  polygon  is  relatively  large  and  symmetrical 
and  thus  would  likely  be  a  good  system  in  terms  of  meeting  functional  requirements. 

A  second  emergent  feature  of  this  view  is  the  cost  distribution.  If  there  is  a 
functional  requirement  that  is  driving  the  cost  of  the  system,  the  fan  representing  that 
requirement  will  be  mostly  black,  while  all  other  fans  will  be  mostly  white.  By  having 
one  fan  different  than  all  other  fans,  it  will  be  more  salient,  thus  giving  the  association 
that  the  difference  should  be  noticed  and  potentially  remedied.  If  the  functional 
requirements  are  balanced  in  cost,  all  the  fans  will  be  mostly  black.  Similar  to  the 
polygon's  shape,  this  natural  mapping  follows  a  user's  appeal  towards  symmetry. 

In  terms  of  data  elements,  the  primary  data  element  is  the  fan  representing  a 
functional  requirement.  Encoded  within  the  fan  is  the  degree  to  which  the  functional 
requirement  is  met,  the  overall  cost  of  the  functional  requirement,  and  the  cost  of  the 
sub-functional  requirements  within  the  fan.  In  this  manner  a  single  data  element  is 
encoding  three  of  the  trade  space  variables  outlined  in  section  2.2.1. 

The  polygonal  element  that  encodes  the  high-level  information  of  how  well  the 
system  meets  all  functional  requirements  is  an  additional  data  element  in  this  view.  The 
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polygonal  area  allows  decision  maker  to  compare  systems  within  the  acquisition  trade 
space;  it  is  not  an  absolute  computation.  For  example,  if  the  functional  requirements 
were  rearranged,  the  system  would  have  a  different  area.  However,  decision  makers  are 
not  obtaining  an  exact  figure  for  how  well  the  functional  requirements  are  being  met, 
they  are  analyzing  the  cost-benefit  trade-offs  between  the  systems.  These  trade-offs  can 
easily  be  completed  with  the  System  View  as  long  as  all  competing  systems  present  the 
functional  requirements  in  the  same  order  around  the  polygon. 

3.3.3.  Multi-System  View 

The  Multi-System  View  displays  two  or  more  system  views  side  by  side,  as  shown 
in  Figure  8.  This  design  is  able  to  directly  promote  comparisons  among  the  systems. 
Because  all  necessary  information  is  positioned  within  the  user's  visual  scan,  the 
decision  maker  benefits  from  uninterrupted  visual  reasoning  [29],  which  allows  the 
decision  makers  to  focus  on  the  differences  between  the  systems  easily  as  all 
information  is  positioned  within  their  visual  range.  The  idea  is  to  emphasize  the 
differences  in  the  data,  and  not  the  manner  which  these  changes  are  being  displayed 
[29].  For  instance.  Figure  8  quickly  reveals  that  the  system  at  the  right  is  much  smaller 
than  those  on  the  left,  but  the  cost  distribution  is  very  similar.  This  type  of  direct 
comparison  could  greatly  help  the  decision  makers  conduct  their  cost-benefit  analysis. 

Decision  makers  can  view  up  to  four  systems  at  a  time  in  the  Multi-System  View. 
However,  the  decision  maker  can  change  which  four  systems  are  being  analyzed  by 
dragging  that  system  from  the  tree  structure  into  one  of  the  four  quadrants.  This 
enables  the  systems  to  remain  sufficiently  large  for  a  user  to  distinguish  the  features  of 
each  system,  while  displaying  multiple  systems  at  a  time. 
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The  emergent  features  of  the  Multi-System  View  are  very  similar  in  nature  to  those 
in  the  System  View,  as  the  same  trade  space  criteria  are  displayed.  The  primary 
difference,  however,  is  that  the  sub-functional  requirement  cost  for  each  system  is  now 
shown  relative  to  the  most  expensive  sub-functional  requirement  for  all  systems  being 
analyzed  in  the  trade  space.  This  allows  the  decision  maker  to  perceive  relative 
differences  among  the  systems  more  readily.  In  this  manner,  decision  makers  can 
compare  sub-functional  requirements,  functional  requirements,  or  full  systems  within 
the  trade  space. 

3.3.4.  Functional  Requirement  View 

The  Functional  Requirement  View  (Figure  9)  displays  multiple  systems  for  a  single 
functional  requirement.  This  view  allows  decision  makers  to  probe  deeper  into  the 
potential  tradeoffs  within  the  trade  space.  It  permits  users  to  view  multiple  systems,  as 
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in  the  Multi-System  View,  but  allows  a  greater  degree  of  detail.  The  decision  makers 
can  analyze  up  to  four  systems  in  this  view  at  a  time,  and  gain  access  to  additional 
systems  by  dragging  the  view  to  the  right  or  the  left.  This  allows  them  to  scroll  through 
all  systems  within  the  system  acquisition  trade  space.  The  decision  makers  can  also  opt 
to  not  view  all  systems  by  dragging  systems  out  of  the  view  back  into  the  tree  structure. 

In  this  view,  the  degree  to  which  the  functional  requirements  are  met  has  been 
modified  from  the  polygonal  structure  to  flat  lines  with  the  delineations:  1)  Does  not 
Meet  Requirements  (bottom  line),  2)  Partially  Meets  Requirements,  3)  Meets 
Requirements  (middle  line,  shown  in  red),  4)  Exceeds  Requirements,  and  5)  Greatly 
Exceeds  Requirements  (top  line).  For  example,  in  Figure  9  the  first  system  greatly 
exceeds  the  requirement,  the  second  only  partially  meets  the  requirement,  and  the  third 
exceeds  the  requirement. 
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3.3.5.  Comparison  View 

The  Comparison  View,  as  shown  in  Figure  10  provides  a  higher  level  of  data 
abstraction  by  displaying  two  or  more  systems  without  the  lower  level  sub-functional 
requirement  cost  information.  This  gives  the  user  the  ability  to  obtain  a  global  view  of 
the  trade  space.  The  polygonal  shapes  each  represent  a  system  in  the  same  manner,  as 
shown  in  the  System  View.  The  fans,  however,  have  been  removed,  deleting 
information  regarding  the  sub-functional  requirement  cost.  Instead,  total  cost  is 
displayed  as  a  function  of  the  color  of  the  system's  polygonal  representation.  The  color 
of  the  polygon  is  determined  by  the  relative  cost  of  a  particular  system  to  the  other 
systems  in  the  trade  space.  A  color  legend  in  the  lower  left  of  Figure  10  displays  both 
the  relative  placement  of  the  systems'  cost  as  well  as  a  digital  value  for  that  cost.  The 
color  gradient  is  an  interval  sequence  which  ranges  from  blue  for  the  most  expensive 
system,  to  yellow  for  the  least  expensive  system  with  all  color  gradations  in  between. 


Figure  10:  Comparison  View  of  FanVis 
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In  this  example,  the  system  with  the  highest  cost,  the  blue  system,  is  not  the  system 
that  most  successfully  achieves  the  functional  requirements.  The  system  with  the 
highest  performance  in  terms  of  the  functional  requirements  is  the  system  with  the 
middle  cost,  the  purple  system.  The  system  with  the  worst  functional  requirement 
performance  is  the  least  expensive  system.  The  Comparison  View  fosters  this  type  of 
cost-benefit  trade-off  analysis. 

As  in  the  System  View,  there  are  several  emergent  features  included  in  the 
Comparison  view  that  are  important  to  note.  As  in  the  previous  views,  the  system  that 
best  meets  requirements  will  have  the  largest  polygonal  shape.  In  Figure  10,  the  system 
that  best  meets  requirements  is  the  one  whose  polygon  encompasses  the  other  two 
polygons,  the  purple  system.  A  second  emergent  property  is  the  determination  of 
relative  cost.  Since  the  cost  scale  for  the  trade  space  is  coded  to  a  color  interval 
sequence,  it  can  be  seen  which  system  has  the  most  expensive  cost,  either  in  total  cost  or 
cost  per  life  cycle  phase. 

3.3.6.  The  "-ility"  View 

Decision  makers  can  obtain  additional  information  in  the  "-ility"  View,  as  shown  in 
Figure  11.  In  this  view,  a  decision  maker  can  analyze  how  well  "-ilities"  are  met  in 
addition  to  analyzing  the  degree  to  which  the  requirements  and  cost  are  met.  How  well 
each  "-ility"  is  met  is  shown  by  scaling  the  size  of  a  system's  polygonal  representation. 
This  scale  is  a  three  point  Likert  scale  ranging  from  1)  Does  not  Meet  Requirements 
(shrinking  the  polygon  from  its  original  size),  2)  Meets  Requirements  (original  size)  and 
3)  Exceeds  Requirements  (expanding  the  polygon  from  its  original  size).  A  three  point 
Likert  scale  is  used  instead  of  the  five  point  Likert  scale  for  how  well  functional 
requirements  are  met  due  to  the  subjective  nature  of  "-ilities."  For  the  most  part,  "- 
ilities"  cannot  be  quantitatively  measured  as  they  are  an  evaluation  of  the  performance 
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of  a  system.  Measuring  them  on  a  finer  scale  could  ultimately  lead  to  data 
misconceptions  by  the  decision  makers  [17]. 

In  this  manner,  a  smaller  polygon  represents  that  an  "-ility"  was  not  being  met.  Thus 
the  optimal  system  would  have  a  large  polygons  for  each  "-ility."  The  predominant 
emergent  features  in  this  view  are  the  polygon's  size  and  symmetry.  Similar  to  the 
Multi- System  View,  the  decision  makers  can  view  up  to  four  systems  at  a  time  in  the  "- 
ility"  View.  To  view  other  systems,  they  can  drag  systems  in  and  out  of  the  four 
columns  and  the  tree  structure. 


Figure  11:  The  "-ility"  View  in  FanVis 

Overall,  each  of  the  tabs  within  FanVis  allows  the  decision  maker  to  gain  a 
different  perspective  on  the  system  acquisition  decision  trade  space.  The  Comparison 
and  "-ility"  Views  allow  decision  makers  to  obtain  a  global  view  of  the  trade  space  data. 
The  System  and  Muti- System  Views  allow  decision  makers  to  inspect  the  system's  cost 
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distribution.  If  this  level  of  detail  is  not  sufficient,  the  decision  makers  can  use  the 
Functional  Requirement  View  to  review  the  data  of  a  single  functional  requirement. 
Each  of  these  perspectives  allows  the  decision  maker  to  assess  different  levels  of  the 
trade  space  data.  This  next  section  describes  the  separable  display,  followed  by  the 
similarities  and  differences  of  the  two  displays. 

3.4.  Separable  Decision  Support  Tool  Design 

A  separable  decision  support  tool  was  also  created  to  test  how  well  the  two  tools 
supported  system  acquisition  decisions.  The  separable  tool  was  created  in  Excel®  since  it 
can  be,  and  likely  is,  a  very  common  application  for  conducting  system  acquisition 
decisions.  The  Excel®  spreadsheet-based  tool  is  a  relatively  simple  decision  support  tool, 
built  entirely  from  functions  within  Excel®.  The  tool  utilizes  four  tabs.  Three  of  the  tabs. 
Requirements  and  "-ilities".  Total  Cost,  and  Cost  Categories,  are  graphical  displays  of 
the  data,  while  the  last  tab.  Data,  includes  the  raw  numbers  of  the  trade  space.  Unlike 
FanVis,  decision  makers  may  experience  difficulty  in  adding  or  deleting  system  data 
dynamically,  as  well  as  reproducing  the  new  charts  automatically  as  these  functions 
must  be  selected  in  Excel®. 

3.4.1.  Requirements  and  "-ilities"  Tab 

The  Requirements  and  "-ilities"  Tab  displays  the  degree  to  which  each  requirement 
is  met  by  systems  in  the  trade  space  in  two  bar  charts.  Each  requirement  or  "-ility"  is 
represented  by  a  different  bar,  while  each  system  has  a  different  color  code  (shown  in 
Figure  12).  This  color  code  was  the  default  Excel®  color  scheme  for  three  variables  in  a 
line  chart.  This  color  scheme  was  retained  as  it  gave  sufficient  separation  among  the 
three  colors. 
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Figure  12:  Requirements  and  "-ilities"  Tab  of  Excel® 

Unlike  FanVis  where  data  elements  were  often  encoded  with  data  from  multiple 
trade  space  variables,  each  data  element  within  the  Excel®  tool  only  represents  a  single 
trade  space  variable.  In  Figure  12,  the  data  elements  in  the  left  bar  chart  each  represent 
how  a  system  meets  an  "-ility."  The  data  elements  in  the  right  bar  chart  each  represent 
how  a  system  meets  a  functional  requirement.  There  are  no  data  elements  which 
represent  how  the  functional  requirements  as  a  whole  are  met.  Instead,  decision  makers 
must  integrate  this  information  themselves. 

3.4.2.  Total  Cost 

The  Total  Cost  Tab  displays  each  system's  total  cost  in  a  bar  chart  as  well  as  the  cost 
per  sub-functional  requirement  in  a  line  chart,  as  shown  in  Figure  13.  A  line  connects 
the  cost  of  the  sub-functional  requirements  within  a  given  functional  requirement. 
These  lines  help  the  decision  maker  delineate  the  various  functional  requirements  from 
each  other  in  any  given  system. 
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Figure  13:  Total  Cost  Tab  of  Excel® 


3.4.3.  Cost  Categories 


The  Cost  Category  Tab  (Figure  14)  is  much  like  the  Total  Cost  Tab.  For  each  life  cycle 
cost  phase  being  analyzed,  the  total  system  cost  appears  for  all  systems  in  a  bar  chart.  In 
addition,  the  cost  per  sub-functional  requirement  for  all  systems  is  shown  in  line  charts 
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Figure  14:  Cost  Categories  Tab  of  Excel® 
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3.4.4.  Data 


The  Data  Tab  (Figure  15)  is  simply  the  raw  trade  space  numbers,  organized  by 
system.  All  trade  space  data  can  be  found  under  this  tab  including  the  degree  to  which 
a  system  meets  the  requirements  and  "-ilities"  as  well  as  the  cost  per  sub-functional 
requirement  for  each  of  the  life  cycle  phases  being  analyzed.  The  Data  Tab  is  organized 
in  such  a  way  that  a  system  acquisition  decision  maker  could  add  additional  functional 
or  sub-functional  requirements.  However,  the  charts  would  not  be  reproduced 
automatically;  they  can  only  be  reproduced  through  the  Excel®  functions. 


Figure  15:  Data  Tab  of  Excel® 

3.5.  Summary 

Both  the  configural  and  separable  display  were  designed  to  encode  the  trade 
space  variables  needed  for  a  system  acquisition  decision.  As  outlined  in  section  2.2.1, 
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these  include  the  degree  functional  requirements  are  met,  the  degree  "-ilities"  are  met, 
and  cost.  Only  the  manner  which  these  trade  space  variables  are  encoded  differs.  In 
FanVis,  most  data  elements  encode  two  or  more  trade  space  variables,  whereas  in  the 
Excel®  tool,  each  data  element  only  encodes  one  trade  space  variable.  The  fans  in 
FanVis,  for  example,  encode  three  trade  space  variables:  the  degree  to  which  the 
functional  requirement  is  being  met,  the  overall  cost  of  the  functional  requirement,  and 
the  cost  of  the  sub-functional  requirements  within  the  fan.  The  functional  requirement 
bars  in  Excel®,  however,  only  encode  one  trade  space  variable:  the  degree  to  which  the 
functional  requirement  is  being  met.  This  allows  FanVis  to  encode  more  information  in 
the  same  space,  while  ensuring  the  decision  maker  is  still  able  to  extract  the  necessary 
information.  In  addition,  by  utilizing  emergent  features,  FanVis  has  data  elements 
which  display  the  higher  level  constraints  of  the  trade  space  such  as  how  the  functional 
requirements  as  a  whole  are  met  (represented  by  the  polygon  area  data  element). 

These  differences  cause  the  two  decision  support  tools  to  have  a  disparate 
number  of  total  number  of  available  data  elements.  For  example,  if  the  trade  space 
consisted  of  three  systems  being  evaluated  on  five  functional  requirements  (fifteen  total 
for  the  data  space)  containing  five  sub-functional  requirements  each  (seventy-five  total 
sub-functional  requirements)  and  four  "-ilities"  (twelve  total),  FanVis  would  have  a 
total  of  278  data  elements  as  compared  to  the  171  data  elements  in  Excel®.  These 
elements  would  be  distributed  within  each  tool  as  illustrated  in  Table  1. 


Table  1:  Data  element  distribution  for  an  example  trade  space 
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Total 
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f.r.  met 

15 

5 

15 

3 

15 

“-ilities”  met 

12 

12 

Total  Cost 

3 

3 

3 

3 

Cost  per 
sub  f.r. 

25 

75 

15 

75 

Life  Cycle 
Cost 

3 

25 

75 

15 

75 
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FanVis  and  the  Excel®  tool  each  have  their  own  set  of  advantages  and 
disadvantages.  FanVis  is  a  series  of  configural  displays,  while  Excel®  is  a  series  of 
separable  displays.  It  is  likely  that  the  configural  displays  will  allow  decision  makers  to 
understand  high-level  information  more  easily  than  in  the  separable  displays,  as  this 
information  is  being  displayed  through  emergent  properties.  Thus,  decision  makers  will 
be  provided  with  more  support  on  Knowledge  Synthesis  and  Information  Aggregation 
tasks  while  using  FanVis.  On  the  other  hand,  when  this  integration  is  not  necessary, 
users  may  find  using  a  separable  display  is  easier  as  the  individual  pieces  of 
information  are  not  already  integrated.  However,  FanVis  was  designed  with  the 
foreknowledge  that  both  integration  and  non-integration  tasks  would  be  necessary, 
thus  the  low-level  information  is  coded  in  a  salient  and  easily  understandable  manner. 
With  these  design  considerations  in  mind,  it  is  believed  that  the  configural  display, 
FanVis,  will  be  able  to  support  a  decision  maker  to  a  similar  or  better  degree  than  a 
separable  display.  This  hypothesis  was  tested  as  outlined  in  the  next  chapter. 
Experiment  Evaluation. 
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4.  Experimental  Evaluation 

Given  the  differences  between  the  two  tools,  the  next  step  in  the  research  process 
was  to  determine  which  tool  best  supported  a  system  acquisition  decision  and  why.  A 
human  performance  experiment  was  conducted  to  compare  the  two  tools  on  a  set  of 
system  acquisition  tasks.  This  chapter  describes  the  experimental  method,  including  the 
setup,  tasks,  and  design  of  the  experiment. 

4.1.  Hypothesis 

It  was  hypothesized  that  the  configural  decision  support  tool,  FanVis,  would  be  able 
to  support  high-level  system  acquisition  decisions  to  a  greater  degree  than  the 
traditional  separable  decision  support  tool  developed  in  Excel®.  This  hypothesis  was 
tested  in  terms  of  participant  performance  on  system  acquisition  tasks,  subjective 
appeal  of  the  decision  support  tools,  and  participants'  cognitive  strategies  while  using 
the  tools. 

4.1.1.  Performance 

It  was  hypothesized  that  the  configural  decision  support  tool,  FanVis,  would  enable 
a  user  to  achieve  higher  performance  compared  to  a  traditional  decision  support  tool. 
Performance  was  measured  in  terms  of  the  percentage  of  system  acquisition  trade  space 
questions  answered  correctly  and  the  speed  at  which  they  were  answered.  These 
questions  were  asked  at  the  three  levels  of  reasoning  a  decision  maker  may  have  to 
utilize  while  conducting  a  system  acquisition  decision  including  Knowledge  Synthesis, 
Information  Aggregation,  and  Data  Processing,  as  defined  in  section  2.2.2. 

The  promotion  of  meta-analysis  through  the  use  of  emergent  features  within 
FanVis  made  it  reasonable  to  expect  that  participants  would  have  increased  accuracy 
for  Knowledge  Synthesis  and  Information  Aggregation  questions  using  FanVis.  The 
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emergent  features  allow  users  to  analyze  the  low-level  information  of  a  trade  space 
(such  as  how  single  functional  requirements  are  met)  and  display  them  in  such  a 
manner  that  higher-level  interactions  are  conveyed  (such  as  how  the  functional 
requirements  as  a  whole  are  met).  This  type  of  meta-analysis  allows  users  to  gain  more 
information  in  a  shorter  amount  of  time.  It  was  expected  that  participants  would 
achieve  similar  accuracy  with  both  tools  on  Data  Processing  questions  since  separable 
displays  have  been  shown  to  support  these  types  of  tasks  to  a  greater  degree  than 
configural  displays.  FanVis  was  designed  in  such  a  way  to  simplify  these  low-level 
tasks  by  displaying  the  low-level  information  as  salient  and  understandable  as  possible. 

In  addition,  participants  were  expected  to  achieve  the  highest  accuracy  on  Data 
Processing  questions  since  answering  these  requires  the  lowest  level  of  reasoning 
(Figure  16).  Furthermore,  it  was  expected  participants  would  answer  a  higher 
percentage  of  Information  Aggregation  questions  correctly  than  Knowledge  Synthesis 
questions  since  the  former  require  a  lower  level  of  reasoning  than  the  latter.  However, 
analyzing  performance  among  the  three  reasoning  levels  was  deemed  to  be  of  less 
importance  than  analyzing  performance  between  the  two  decision  support  tools. 


Average 
Percentage 
of  Correct 
Answers 


Reasoning  Difficulty  Level 

Figure  16:  Hypothesized  results  for  percentage  of  correct  decision  choices 
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Participants  were  expected  to  be  able  to  analyze  the  trade  space  in  a  more  efficient 
manner  while  answering  Knowledge  Synthesis  and  Information  Aggregation  questions 
using  FanVis.  As  mentioned  above,  because  FanVis  allows  users  to  conduct  meta¬ 
analyses  while  analyzing  the  trade  space,  it  was  expected  that  the  participants  could  be 
able  to  obtain  more  information  in  a  shorter  time  span.  While  users  of  configural 
displays  have  been  shown  to  have  less  efficient  performance  for  low-level  data 
extraction  in  some  studies  [30],  the  low-level  data  in  FanVis  has  been  organized  in  such 
a  way  to  aid  users  extract  this  type  of  data.  For  this  reason  it  was  expected  that 
participants  would  answer  the  Data  Processing  questions  with  a  similar  speed  with 
FanVis  as  the  separable  Excel®  tool  (Figure  17).  It  was  also  expected  that  participants 
would  spend  a  greater  amount  of  time  on  questions  requiring  the  greatest  amount  of 
reasoning,  and  thus  would  answer  Data  Processing  questions  in  the  shortest  time  frame, 
followed  by  Information  Aggregation  questions  and  Knowledge  Synthesis  questions. 


Average 

Time 


Figure  17:  Hypothesized  results  for  time  to  answer  questions 
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4.1.2.  Subjective  Appeal 

Subjectively,  it  was  expected  that  users  would  prefer  using  FanVis  over  the  Excel®- 
based  separable  tool  while  completing  a  system  acquisition  decision  since  FanVis 
presents  the  information  in  a  clearer  and  more  interactive  manner.  However,  it  was 
acknowledged  that  there  potentially  could  be  a  bias  towards  the  Excel®  tool,  as  the 
majority  of  the  participants  were  proficient  with  Excel®  whereas  FanVis  was  a  new  tool 
for  all  participants.  This  bias  and  its  potential  implications  to  this  study  are  discussed 
further  in  the  Corollary  Hypothesis  section.  The  user's  subjective  appeal  was 
determined  by  analyzing  the  participants'  responses  to  a  questionnaire  at  the  end  of  the 
experiment. 

4.1.3.  Cognitive  Strategies 

Cognitive  strategies  of  the  participants  were  also  analyzed  through  the  use  of  eye 
fixation  patterns  to  help  determine  if  the  participants  were  accessing  the 
relevant/necessary  elements  to  complete  their  decision  process,  and  if  so,  how 
efficiently  they  were  accessed.  An  efficient  eye  fixation  pattern  is  a  natural  fixation 
pattern  where  the  shortest  path  is  taken  from  one  necessary  element  to  the  next.  Natural 
fixation  patterns  suggest  participants  utilized  superior  cognitive  strategies  compared  to 
if  they  were  using  a  less  efficient  gaze  pattern  [31-33].  The  fixations  were  obtained  by 
gathering  data  from  an  eye  tracking  device  (discussed  in  section  4.3.2).  These  fixations 
identified  which  elements  participants  were  focusing  on  to  a  greater  degree  in  each  of 
the  displays,  and  what  fixation  patterns  emerged  through  the  use  of  each  tool.  It  was 
believed  that  participants  would  be  able  to  access  the  necessary  elements  in  a  more 
efficient  manner  while  using  FanVis  as  compared  to  the  Excel®  tool  since  the  trade  space 
variables  were  encoded  in  a  more  effective  manner  within  FanVis's  data  elements. 
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4.1.4.  Corollary  Hypothesis 

For  all  research  hypotheses  posed,  it  was  acknowledged  that  users  could  become 
confused,  overloaded  or  even  misguided  because  of  their  unfamiliarity  with  the  FanVis 
tool.  It  should  be  noted  that  the  majority  of  participants  in  this  experiment  were 
moderately  to  extremely  proficient  in  Excel®,  whereas  none  had  any  previous 
experience  with  FanVis.  This  bias  could  have  led  to  improved  participant  performance 
with  the  Excel®  tool,  even  if  FanVis  represented  the  trade  space  in  a  more  salient 
manner.  It  was  hypothesized  though,  that  because  the  data  in  FanVis  is  mapped  more 
directly  to  a  user's  cognitive  model  of  the  trade  space,  users  would  be  able  to  readily 
understand  the  data  aspects  being  presented  in  the  tool. 

4.2.  Participants 

To  test  these  hypotheses,  30  participants  between  the  age  of  18  and  75  were 
recruited  for  this  study.  Further,  personnel  with  experience  in  either  high-level  system 
acquisitions  or  high-level  decisions  for  a  team  or  organization  were  specifically 
recruited  as  both  roles  utilize  the  high-level  data  analysis  skills  required  for  an 
acquisition.  Those  with  only  high-level  team  decision-making  experience  differed  in 
that  they  had  not  completed  an  actual  acquisition. 

The  average  participant  age  was  52.43  years  with  a  standard  deviation  of  11.39 
years.  Half  the  participants  had  served  or  were  currently  serving  in  the  armed  forces 
with  an  average  of  16.21  years  of  service  and  standard  deviation  of  13.03  years.  All  had 
moderate  to  high  levels  of  experience  completing  high-level  decisions  for  a  team, 
project  or  organization,  and  23  of  the  30  had  system  acquisition  experience.  The  average 
number  of  years  of  system  acquisition  experience  of  those  participants  was  13.95  years 
with  a  standard  deviation  of  13.10  years.  All  had  experience  using  data  manipulation 
tools,  such  as  Excel®,  and  none  indicated  that  they  were  color  blind.  All  participants 
could  comfortably  see  the  information  presented  on  the  computer  screens  for  the 
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duration  of  the  experiment.  The  full  demographic  information  of  the  participants  can  be 
found  in  Appendix  A. 

4.3.  Apparatus 

Two  main  pieces  of  equipment  were  required  to  complete  this  study;  the  Mobile 
Advanced  Command  and  Control  Station  (MACCS)  and  an  eye  tracking  system.  This 
section  outlines  both  pieces  of  equipment,  how  they  were  used  within  the  context  of 
this  experiment,  and  how  they  contributed  to  the  experiment  as  a  whole. 

4.3.1.  MACCS 

The  Mobile  Advanced  Command  and  Control  Station's  an  experimental  test  bed 
equipped  with  six  21-inch  wall  mounted  displays,  each  having  a  resolution  of 
1280x1024  pixels,  16  bit  color  resolution.  The  displays  are  organized  as  shown  in  Figure 
18.  For  the  purposes  of  this  experiment,  only  the  bottom  three  monitors  were  used 
while  the  top  three  were  not  powered.  The  computer  used  to  run  the  simulator  was  a 
Digital  Tiger  Stratosphere  Titan  with  an  AMD  Athlon  64  X2  Duel  Core  Processor  4200+  and 
four  NVIDIA  Quadro  NVS  285  graphics  cards. 


Figure  18:  Inside  view  of  the  Mobile  Advanced  Command  and  Control  Station 
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MACCS  is  a  mobile  testing  platform  mounted  within  a  2006  Dodge  Sprinter 
shown  in  Figure  19.  By  integrating  an  experimental  test  bed  into  a  vehicle,  the 
experiment  was  able  to  travel  to  the  participants,  making  the  experimental  process 
easier  for  the  participant.  This  allowed  a  high  number  of  participants  to  be  recruited 
with  system  acquisition  or  high-level  decision-making  knowledge. 


Figure  19:  Outside  view  of  the  Mobile  Advanced  Command  and  Control  Station 


4.3.2.  Eye  Tracker 

An  eye  tracker  was  used  to  collect  the  participants'  eye  fixation  data  as  they 
answered  the  system  acquisition  trade  space  questions.  As  mentioned  in  the  hypotheses 
section,  analyzing  the  fixation  data  helped  determine  the  cognitive  strategies  of  the 
participants.  The  eye  tracker  used  was  a  Polhemus  VisionTrak®  [34],  As  shown  in 
Figure  20,  this  is  a  head-mounted  eye  tracking  system  on  a  baseball  cap  with  an 
adjustable  head  band.  A  baseball  cap  was  chosen  to  minimize  subject  discomfort  and 
allow  full  head  movement. 

The  eye  tracking  system  tracks  the  center  of  a  participant's  pupil  and  the  reflection 
from  the  corneal  surface.  This  tracking  information  is  integrated  with  head  movement 
data  to  determine  the  fixation  point  of  the  participant.  This  technology  was  developed 
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with  the  Polhemus  VisionTrack®  System  by  ISCAN®  [35].  The  head  movement  data  is 
found  using  a  magnetic  source  and  sink.  The  magnetic  source  measures  one  inch  square 
and  is  located  approximately  two  feet  from  the  participant,  as  can  be  seen  in  Figure  18. 
The  magnetic  sink  is  a  small  gray  attachment  on  the  brim  of  the  baseball  cap  as  shown 
in  Figure  20. 


Figure  20:  The  Polhemus  VisionTrack®  System 


Fixation  points  are  mapped  to  the  computer  screen  to  produce  real-time  vision 
tracking.  This  allows  the  participant  to  browse  over  a  large  area  through  the  duration  of 
the  experiment.  It  also  allows  the  participant  to  look  down,  or  away  from  the  screens  of 
interest,  and  be  able  to  return  to  the  screen.  A  simple  calibration  routine  was  completed 
at  the  beginning  of  the  experiment  to  accommodate  for  each  participant's  specific  head 
and  body  orientation  to  the  displays. 


4.4.  Experimental  Procedure 

The  experiment  consisted  of  seven  parts:  pre-experiment  interactions,  a  baseline 
data  handling  proficiency  test,  two  training  sessions,  two  test  sessions,  and  post- 
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experiment  interactions.  This  section  discusses  the  components  of  each  of  these  seven 
parts.  On  average,  the  full  experiment  lasted  an  hour  and  a  half. 

4.4.1.  Pre-Experiment  Interactions 

When  participants  arrived,  they  were  introduced  to  the  experiment  as  well  as  the 
experimental  setup.  The  participants  then  read  and  signed  the  Consent  to  Participate 
Form  in  which  they  learned  the  purpose  of  the  experiment,  compensation  conditions 
and  the  experimental  aspects  they  were  asked  to  complete  (Appendix  B).  The 
participant  then  filled  out  a  brief  demographic  survey  (Appendix  C). 

4.4.2.  Baseline  Data  Handling  Proficiency  Test 

A  baseline  data  handling  proficiency  test  was  administered  to  assess  participants' 
Excel®  familiarity  and  data  processing  skills.  This  test  was  constructed  from  the 
Educational  Testing  Service®  practice  questions  for  the  quantitative  section  of  the 
Graduate  Record  Examinations  [36,  37],  The  test  consisted  of  quantitative  multiple 
choice  questions  that  were  answered  by  interpreting  Excel®  charts  and  graphs.  The 
questions  had  a  varying  degree  of  difficulty,  with  a  definitive  answer  (Appendix  D). 
The  participants'  baseline  data  handling  proficiency  was  based  upon  the  percentage  of 
questions  answered  correctly,  and  how  long  the  participants  took  to  answer  all 
questions. 

4.4.3.  Training  Session 

The  participants  were  introduced  to  an  acquisition  case  study  and  then  presented 
with  a  tutorial  of  the  decision  support  tool,  both  of  which  provided  the  participants 
with  the  information  necessary  to  complete  the  system  acquisition  comparison  test 
session.  In  FanVis,  a  case  study  regarding  the  selection  of  a  student  Unmanned  Aerial 
Vehicle  (UAV)  system  by  a  funding  agency  was  presented,  while  in  the  Excel®  tool,  a 
case  study  regarding  laptop  selection  by  a  board  member  of  a  low-income  school 
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district  was  used,  (Appendix  E).  These  two  case  studies  were  built  from  the  same  trade 
space  data  in  order  to  ensure  that  the  two  test  sessions  were  similar  in  difficulty. 

The  trade  space  data  was  obtained  from  the  2007  Association  for  Unmanned 
Vehicle  Systems  International  (AUVSI)  student  competition.  In  this  competition, 
competing  UAVs  had  to  launch,  follow  a  course,  identify  (ID)  objects,  track  moving 
objects,  and  land  successfully  [38].  As  part  of  the  competition,  each  student  team  had  to 
write  a  report  regarding  their  system's  design,  the  accuracy  of  the  system,  and  other 
system  details.  Three  systems  were  chosen  for  use  in  this  experiment's  case  study  based 
upon  how  well  they  performed  in  the  competition  (the  two  best  performing,  and  the 
worst  performing).  The  reports  of  these  systems  were  then  analyzed  to  obtain  the 
system  cost  and  performance  information.  This  trade  space  data  was  presented  as  the 
system  acquisition  trade  space  data  in  FanVis.  In  the  case  study,  the  three  systems  were 
renamed  School  A,  School  B,  and  School  C  to  avoid  any  biases  to  the  schools.  The  three 
systems  were  analyzed  on  seven  functional  requirements  (Launch,  Transition  to 
Autonomy,  Maintain  Flight,  Navigate  Course,  ID  Objects,  Track  Objects,  and  Land)  and 
three  "-ilities"  (Adaptability,  Reliability,  and  Sustainability). 

For  the  Excel®  case  study,  the  trade  space  cost  data  was  scaled  by  0.035  to  more 
accurately  represent  the  costs  of  low-cost  laptops  and  the  order  which  the  functional 
requirements  were  presented  was  rearranged  so  that  it  would  appear  as  if  there  were 
two  entirely  different  trade  spaces  being  presented.  For  example,  the  most  expensive 
functional  requirement  was  presented  as  the  fifth  functional  requirement  in  the  FanVis 
case  study  and  second  in  the  Excel®  case  study.  In  this  manner,  the  same  data  trade 
space  was  presented  for  both  tools,  while  only  the  labels  of  this  data  and  the  manner  it 
was  presented  differed.  In  the  Excel®  case  study,  participants  evaluated  System  1, 
System  2,  and  System  3  on  seven  functional  requirements(Support  State  Setting,  Secure 
Data,  GUI  Plat-formed,  File  Manipulation,  Content  Manipulation,  Information  Sharing, 
and  Modify  Hardware)  and  three  "-ilities"  (Usability,  Reliability,  and  Modularity) 
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Along  with  the  case  study  data,  participants  were  given  the  selection  criteria 
necessary  to  complete  the  system  acquisition.  The  system  selection  criteria  stated  the 
high-level  objectives  that  must  be  met  by  the  selected  system.  In  decreasing  order  of 
importance,  they  were: 

•  At  least  meet  all  "-ilities" 

•  At  least  meet  all  functional  requirements 

•  Minimize  cost 

•  Maximize  degree  "-ilities"  and  functional  requirements  met 

•  Balance 

o  "-ilities"  across  system 
o  Functional  requirements  across  system 
o  Cost  across  sub-functional  requirements 
The  tutorial  (Appendix  F)  gave  participants  an  overview  of  the  decision  support 
tool,  how  data  was  encoded  within  the  tool,  and  specific  features  that  would  likely  be 
necessary  to  utilize  while  completing  the  test  session.  During  the  tutorial,  the 
participants  were  able  to  see  and  interact  with  the  tool  using  a  practice  data  set  and 
were  encouraged  to  ask  questions.  Participants  spent  an  average  of  ten  minutes  on  the 
tutorial.  For  FanVis,  this  was  the  only  time  the  participants  had  to  interact  with  the  tool 
before  the  experiment  began.  For  Excel®,  participants  already  had  a  moderate  to  high- 
level  of  experience  using  the  tool.  Participants  were  encouraged  to  practice  using  both 
tools  until  they  felt  comfortable  with  their  use. 

Following  the  tutorial,  the  eye  tracker  was  calibrated.  This  calibration  ensured 
that  the  correct  data  was  collected  during  the  actual  testing  phase  when  the  participants 
completed  the  experimental  questions. 
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4.4.4.  Test  Session 


In  each  test  session,  the  participant  answered  questions  regarding  the  system 
acquisition  trade  space  described  in  the  case  study.  The  participants  were  asked 
identical  (in  both  format  and  difficulty)  trade  space  questions  in  both  tools.  These 
questions  began  with  six  Knowledge  Synthesis  questions,  followed  by  six  Information 
Aggregation  questions,  six  Data  Processing  questions  and  concluded  with  a  repeat  of 
the  initial  Knowledge  Synthesis  questions.  The  first  and  last  question  was  "Which 
system  best  meets  the  baseline  system  selection  criteria?"  The  questions  were  presented 
in  this  order  and  not  randomized  since  system  acquisition  decision  makers  do  not 
generally  attempt  to  determine  the  best  system  based  on  a  set  of  objective  criteria 
randomly.  Interviews  with  these  decision  makers  demonstrated  that  they  typically 
started  with  broad,  more  ambiguous  questions,  and  then  drilled  down  through 
hierarchical  levels  of  information  to  obtain  answers.  Thus  we  attempted  to  emulate  this 
strategy  through  the  specific  ordering  of  questions,  which  was  held  constant  for  each 
subject.  The  last  question  was  repeated  to  determine  if  the  exploration  of  the  data  space, 
held  constant  for  everyone,  changed  the  participants'  final  decisions. 

All  questions  had  a  definitive  correct  answer.  For  each  question,  four  choices 
were  presented  to  the  participant.  The  list  of  all  questions  asked,  the  possible  choices 
and  the  correct  answer  are  included  in  Appendix  G.  These  questions  were  displayed  on 
a  Graphical  User  Interface  (GUI)  on  the  right  screen  of  the  MACCS,  while  the  decision 
support  tool  loaded  with  the  case  study  data  was  displayed  on  the  center  screen.  The 
system  selection  criteria  list  was  displayed  on  the  left  screen. 

The  first  test  session  was  completed  when  the  participants  answered  all  19 
experimental  questions.  At  this  point,  they  were  offered  a  break  before  continuing  with 
the  next  training  and  test  session.  Participants  completed  two  test  sessions;  one  with 
Excel®  and  the  other  with  FanVis.  The  order  of  these  test  sessions  was  counterbalanced 
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and  randomized.  Each  test  session  was  preceded  by  the  training  session  of  the  tool  to  be 
used  in  that  particular  test  session.  For  example,  participant  number  one  completed  the 
experiment  as  follows:  pre-experiment  interactions,  baseline  data  handling  proficiency 
test,  training  with  Excel®,  testing  with  Excel®,  break,  training  with  FanVis,  testing  with 
FanVis,  and  post-experiment  interactions. 

4.4.5.  Post-experiment  interactions 

A  brief  retrospective  protocol  was  conducted  following  completion  of  both  test 
sessions.  The  intention  of  this  portion  of  the  experiment  was  to  obtain  information 
regarding  why  a  participant  manipulated  the  tools  in  a  specific  manner  and  to  gain  the 
participant's  general  impressions  of  the  tools.  The  general  questions  asked  during  the 
retrospective  verbal  protocol  are  listed  in  Appendix  H,  but  varied  based  upon 
participants'  responses. 

In  addition,  the  participants  were  asked  which  tool  they  felt  was  more  useful,  which 
tool  they  felt  was  more  pleasant  to  use,  which  tool  they  would  prefer  to  use  in  the 
future,  as  well  as  which  tool  they  felt  gave  them  a  better  understanding  of  the  system 
acquisition  trade  space  (Appendix  I).  Finally,  the  participants  were  asked  to  voice  any 
lingering  questions  or  final  thoughts  on  the  displays  or  the  experiment  in  general. 

4.5.  Experimental  Design 

The  experiment  was  a  2x3  repeated  measures  design  with  two  independent 
variables:  Decision  Support  Tool  (FanVis,  Excel®)  and  Reasoning  Difficulty  Fevel 
(Knowledge  Synthesis,  Information  Aggregation,  Data  Processing).  All  participants 
received  all  six  treatment  combinations.  The  order  that  the  participants  received  the  two 
levels  of  Decision  Support  Tool  was  counterbalanced  and  randomly  assigned  to  each 
participant.  The  Reasoning  Difficulty  Fevel  was  presented  in  the  same  order  for  all 
participants,  as  previously  discussed. 
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4.5.1.  Dependent  Variables 

A  number  of  dependent  variables  were  chosen  to  determine  if  the  hypotheses  given 
in  section  4.1  were  correct.  This  section  describes  how  each  of  these  dependent  variables 
was  calculated.  The  results  for  each  dependent  variable  will  be  described  in  Chapter  5. 

•  Score:  Score  is  the  percentage  of  correct  answers  over  the  total  number  of 
questions  within  a  Reasoning  Difficulty  Level.  Thus,  each  participant  has  6  score 
values:  Excel®-Knowledge,  Excel®-Information,  Excel®-Data,  FanVis-Knowledge, 
FanVis-Information,  and  FanVis-Data. 

•  Time  to  Answer:  Time  to  answer  is  the  participant's  cumulative  time  to  answer 
all  questions  within  a  Reasoning  Difficulty  Level. 

•  Subjective  Tool  Preference:  The  subjective  tool  preference  is  a  participant's 
response  to  the  preferred  tool  selection  questions.  As  there  are  four  questions  in 
total,  there  are  four  subjective  tool  preference  values  per  participant. 

•  Percent  of  Time  Fixating  on  Relevant  Elements:  The  percent  time  spent  fixating 
on  relevant  elements  gives  insight  as  to  how  accessible  and  understandable  the 
trade  space  data  is  for  both  the  tools  [39].  For  example,  for  a  question  regarding 
the  functional  requirements,  such  as  "Which  system  meets  the  functional 
requirements  to  the  greatest  degree?",  all  fixations  on  elements  containing  the 
"functional  requirement  met"  element  would  be  considered  relevant,  while  all 
other  elements  would  not  be  considered  irrelevant.  Thus,  for  each  question  there 
is  a  distinct  set  of  elements  that  are  relevant  to  answering  the  question.  The 
percentage  of  time  is  used  as  a  metric  as  opposed  to  the  percentage  of  relevant 
element  fixations  due  to  the  disparate  number  of  elements  within  the  two  tools. 

4.5.2.  Covariates 

In  statistical  analysis,  the  covariate  Proficiency  Time  was  used.  Proficiency  time  was 
the  total  amount  of  time  it  took  the  participant  to  answer  all  eight  questions  within  the 
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baseline  proficiency  test.  In  this  study,  proficiency  time  was  used  to  help  predict  how 
long  it  would  take  the  participants  to  complete  the  system  acquisition  questions  as 
proficiency  time  was  both  a  baseline  for  a  participant's  Excel®  proficiency  and  data 
handling  ability. 
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5.  Results 


This  chapter  presents  the  statistical  results  of  the  experiment  described  in  Chapter  4. 
For  statistical  analysis,  a  2x3  repeated  measures  mixed  linear  model  was  applied  to 
analyze  the  dependent  variable.  Time.  For  all  other  dependent  variables,  non- 
parametric  tests  were  used  since  parametric  assumptions  were  not  met.  An  a  level  of 
0.05  for  all  statistical  tests  was  used.  Additionally,  though  eye  tracker  data  was  gathered 
for  all  thirty  participants,  only  five  participants  (3,  6,  16,  23,  and  24)  had  tracks  which 
were  continuously  accurate  for  the  duration  of  both  the  FanVis  and  Excel®  test  sessions. 
Eye-track  data  was  therefore  analyzed  using  only  these  five  participants. 

5.1.  Score 

As  expected,  a  participant's  score  varied  depending  upon  the  reasoning 
difficulty  level.  As  shown  in  Figure  21,  participants  obtained  a  higher  score  using 
FanVis  than  with  Excel®  when  answering  Knowledge  Synthesis  questions  (Mann- 
Whitney  Dependent  Test  [40],  z=1.99,  p=0.046),  while  participants  obtained  a  higher 
score  using  Excel®  than  with  FanVis  while  answering  Data  Processing  questions  (z=2.21, 
p=0.027).  There  was  no  statistical  difference  between  the  two  decision  support  tools  for 
score  on  Information  Aggregation  questions  (z=0.77,  p=0.437). 

In  addition,  there  was  no  statistical  difference  given  the  order  participants  used 
the  two  tools.  Analyzing  score  by  test  session  using  Mann-Whitney  Dependent  Tests 
[40],  there  was  no  statistical  significance  for  score  on  Knowledge  Synthesis  questions 
(z=-0.62,  p=0.52).  Information  Aggregation  questions  (z=-0.71,  p=0.48)  or  Data 
Processing  questions  (z=-0.40,  p=0.69).  This  indicates  the  participants'  accuracy  on  the 
system  acquisition  questions  was  not  affected  by  the  order  of  the  test  sessions. 


69 


Figure  21:  Average  percentage  of  correct  answers 
Each  participant's  score  performance  was  then  analyzed  on  a  per-question  basis 
to  determine  the  performance  differential  that  arose  when  a  participant  answered  a 
specific  question  correctly  with  one  tool  and  incorrectly  with  the  other.  Because  an 
identical  set  of  questions  asked  for  both  tools,  this  metric  indicates  if  a  participant  was 
only  able  to  extract  the  required  information  out  of  one  of  the  two  tools.  For  instance,  if 
a  participant  answered  a  question  correctly  while  using  Excel®  but  the  same  question 
incorrectly  using  FanVis,  then  there  is  a  performance  differential  in  favor  of  Excel®.  To 
determine  the  total  performance  differential  by  question,  each  participant's 
performance  differential  is  summed  by  question  yielding  the  results  shown  in  Figure 
22.  In  this  manner,  if  all  thirty  participants  answered  a  question  correctly  in  FanVis  and 
incorrectly  in  Excel®,  than  that  tool  would  have  a  thirty  point  performance  differential 
in  favor  of  FanVis.  For  example,  five  participants  answered  question  number  2  correctly 
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in  Excel®  and  incorrectly  in  FanVis,  while  eight  participants  answered  the  question 
correctly  in  FanVis  and  incorrectly  in  Excel®.  Hence,  there  is  a  total  performance 
differential  of  three  points  in  favor  of  FanVis. 


Performance  Differential  by  Question 


o 

o 

X 

111 

</> 

> 

c 

(0 


c 

0) 

a— 

d> 


V 

u 

c 

(0 

E 


a> 

Cl 


Question 


Figure  22:  Total  performance  differential  by  question 
The  performance  differential  echoes  the  performance  score  results  overall  for  the 
three  Reasoning  Difficulty  Fevels,  as  there  is  a  total  performance  differential  of  17 
points  in  favor  of  FanVis  on  Knowledge  Synthesis  questions,  a  5  point  performance 
differential  in  favor  of  Excel®  on  Information  Aggregation  questions  and  a  10  point 
performance  differential  in  favor  of  Excel®  on  Data  Processing  questions.  This  echoes 
the  general  trend  that  FanVis  provided  superior  performance  for  Knowledge  Synthesis 
tasks,  the  two  tools  provided  similar  performance  for  Information  Aggregation  tasks, 
and  Excel®  provided  superior  performance  for  Data  Processing  tasks. 

What  is  interesting  to  note  are  those  questions  which  have  a  large  differential. 
Questions  3,  6  and  7  have  particularly  high  performance  differentials  in  favor  of  FanVis. 
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These  questions  all  asked  the  participant  to  integrate  the  cost  of  the  sub-functional 
requirements.  This  would  indicate  that  FanVis  allowed  participants  to  complete  this 
task  better  than  in  Excel®.  On  the  other  hand,  questions  9  and  15  have  particularly  high 
performance  differentials  in  favor  of  Excel®.  These  two  questions  asked  the  participants 
to  extract  the  "-ility"  data.  This  would  indicate  that  Excel®  allowed  participants  to 
complete  this  task  better  than  FanVis.  These  findings  will  have  important  ramifications 
on  the  design  of  the  decision  support  tool,  as  will  be  discussed  in  Chapter  6. 

Furthermore,  the  participants'  performance  differential  on  the  question  "Which 
system  best  meets  the  baseline  system  selection  criteria?"  was  analyzed  to  determine  if 
exploring  the  data  space  changed  the  participants'  decisions  as  per  which  system  best 
met  these  criteria.  This  particular  question  was  both  the  first  and  last  question  answered 
by  the  participants.  In  FanVis,  2  participants  answered  the  question  incorrectly  both 
times,  1  initially  answered  correct  but  answered  incorrectly  the  second  time  the 
question  was  asked,  6  initially  answered  incorrectly  but  then  answered  correctly  the 
second  time,  and  21  answered  correctly  both  times.  In  all,  a  marginally  statistically 
significant1  portion  of  the  participants  changed  their  answers  (z=-1.890,  p=0.059), 
suggesting  interaction  with  FanVis  allowed  the  participants  to  obtain  a  clearer  picture 
of  the  system  acquisition  trade  space  over  the  time  frame  of  the  experiment. 

Within  Excel®,  1  participant  answered  correct  but  answered  incorrectly  the 
second  time  the  question  was  asked,  4  initially  answered  incorrectly  but  then  answered 
correctly  the  second  time,  and  25  correctly  answered  the  question  both  times.  In  all,  a 
non-significant  portion  of  the  participants  changed  their  answers  (z=-1.342,  p=0.180). 
Further,  participants  did  not  have  a  statistically  significant  difference  in  their  change 
strategy  when  completing  their  first  test  session  compared  to  their  second  test  session 
(z=-0.513,  p=0.608).  As  the  test  session  order  (FanVis  presented  first  or  Excel®  presented 


1  An  a  value  between  0.05  and  0.10 
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first)  was  counterbalanced  and  randomly  assigned,  this  indicates  participants  answered 
these  questions  based  upon  the  case  study  being  analyzed  for  that  test  session  and  were 
not  affected  by  fact  that  the  case  study  questions  were  asked  in  the  same  manner  for 
both  test  sessions. 

5.2.  Time  to  Answer 

A  logarithmic  transformation  (natural  log)  of  the  dependent  variable  time  to 
answer  was  utilized  to  satisfy  normality  and  homogeneity  assumptions  [40].  A  Levene 
test  indicated  the  homogeneity  assumption  was  not  met  (F(5,172)=15.347,  p<0.001) 
before  the  logarithmic  transformation  was  applied,  but  that  the  homogeneity 
assumption  was  met  after  the  transformation  was  applied  (F(5,172)=0.568,  p=0.724).  In 
addition,  two  outliers  (greater  than  3.29  standard  deviations  from  the  mean)  were 
deleted  from  the  data  set.  With  the  filtered  data  set,  a  2x3  mixed  linear  model,  a=0.05, 
was  used  to  determine  if  Reasoning  Difficulty  Level  or  Decision  Support  Tool  had  a 
significant  effect  on  time  to  answer  (Appendix  J).  It  was  found  that  the  proficiency  time 
metric  had  a  positive  correlation  to  the  log  of  time  to  answer  through  the  Spearman 
Rank  Correlation  (p  =0.271,  p<0.001).  Proficiency  time,  as  described  in  Section  4.5.2,  was 
the  time  it  took  the  participants  to  answer  a  set  of  baseline  data  processing  questions 
based  on  Excel®  charts  and  graphs.  Thus,  proficiency  time  was  included  as  a  covariate. 
The  resultant  model  is  shown  Equation  (1). 

Yijkl  =  A-  +  A  +  +  Pk  +  (ap)jk  +  (pa)tJ  +  (p/3)lk  +  y(XijU )  +  em  (1) 

In  this  equation/^  is  a  constant,  pi  are  the  participants,  a j  are  the  Decision 
Support  Tools,  f3k  are  the  Reasoning  Difficulty  Levels,  {a/3)  jk  are  the  interaction  effects 
between  Decision  Support  Tools  and  Reasoning  Difficulty  Levels,  (palmare  interaction 
effects  between  the  participants  and  the  Decision  Support  Tools,  (p/3)jk  are  the 
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interaction  effects  between  the  participants  and  the  Reasoning  Difficulty  Levels,  y{Xjjkl) 
are  the  effects  adjustments  from  the  covariate  and  eijkl  are  the  residual  errors. 

Using  this  model,  proficiency  time  was  found  to  be  significant  (F(l,56)=21.81, 
p<0.001).  In  addition,  a  significant  difference  was  found  for  Decision  Support  Tool 
(F(l,29)=  12.17,  p=0.0016)  and  Reasoning  Difficulty  Level  (F(l,29)=216.71,  p<0.001).  Most 
of  the  differences  among  the  six  treatments  were  statistically  significant  when  analyzed 
as  pair  wise  comparisons,  shown  in  Table  2.  However,  three  points  were  not  statistically 
significant:  Excel®-Information/FanVis-Data  (p=0.251),  Excel®-Information/F an  Vis- 
Information  (p=0.032),  and  Excel®-Knowledge/FanVis-Knowledge  (p=0.471).  It  is  the 
two  latter  pairs  which  are  most  interesting,  as  this  indicates  there  was  no  statistical 
difference  in  time  between  the  two  tools  to  complete  Knowledge  Synthesis  and 
Information  Aggregation  questions,  as  can  be  seen  in  Figure  23. 


Table  2:  Pair  wise  comparisons  with  significant  differences  in  highlighted  cells 


Excel® 

FanVis 

Knowledge 

Information 

Data 

Knowledge 

Information 

Data 

Excel® 

Knowledge 

<0.001 

<0.001 

0.471 

<0.001 

<0.001 

Information 

<0.001 

<0.001 

0.032 

0.251 

Data 

<0.001 

0.002 

<0.001 

FanVis 

Knowledge 

<0.001 

<0.001 

Information 

<0.001 

Data 

Figure  23  yields  insight  into  the  actual  average  time  participants  spent  to  answer 
the  system  acquisition  decision  support  questions.  The  greatest  time  difference  between 
the  two  decision  support  tools  on  a  given  reasoning  difficulty  level  occurred  on  Data 
Processing  questions.  Here,  participants  spent  an  average  of  60.8  seconds  more  in 
FanVis  than  Excel®.  Though  this  comparison  was  found  to  be  statistically  significant,  it 
is  a  little  more  than  one  minute,  altogether  a  small  amount  of  time.  Further,  time  is  not 
as  important  as  accuracy  for  system  acquisition  decisions.  While  it  is  important  to  note 
the  statistical  significance,  for  an  actual  decision,  this  minute  would  likely  not  matter.  In 
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addition,  participants  may  have  spent  a  greater  amount  of  time  answering  the  system 
acquisition  decision  support  questions  in  FanVis  simply  due  to  the  greater  number  of 
data  elements  within  the  tool.  As  mentioned  in  section  3.5,  FanVis  contains  a  total  of  278 
data  elements  whereas  Excel®  contains  171  data  elements.  Accessing  these  additional 
hundred  elements  could  have  certainly  caused  this  time  disparity.  Future  research 
could  analyze  if  this  hypothesis  is  correct,  or  if  there  are  different  motivating  factors. 

Average  Time  to  Complete  Questions  by  Reasoning 
Difficulty  Level 


Decision 
Support  Tool 

— ♦ — FanVis 
-  -  •  -  -  Excel 


Reasoning  Difficulty  Level 

Figure  23:  Average  time  to  answer  questions 

Similar  to  score,  there  was  no  statistical  difference  given  the  order  participants 
used  the  two  tools.  Analyzing  time  by  test  session  and  Reasoning  Difficulty  level  using 
a  2X2  Analysis  of  Variance  (Appendix  J),  test  session  was  not  statistically  significant 
(F(l,172)=0.79,  p=0.373).  This  indicates  the  participants'  response  times  to  the  system 
acquisition  questions  were  not  effected  by  the  order  of  the  test  session. 
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5.3.  Subjective  Tool  Preference 

Subjective  tool  preference  was  found  by  analyzing  the  participants  responses  to  a 
set  of  subjective  questions  using  a  Mann-Whitney  Dependent  Test  [40].  A  statistically 
significant  portion  of  the  participants  felt  FanVis  was  a  more  useful  tool  than  Excel® 
(z=2.01,  p=0.04),  felt  that  FanVis  was  able  to  give  them  a  better  understanding  of  the 
trade  space  than  Excel®  (z=3.10,  p=0.002),  and  would  choose  to  use  FanVis  over  Excel® 
given  the  opportunity  during  their  next  system  acquisition  decision  (z=2.01,  p=0.04).  A 
marginally  statistically  significant  portion  of  the  participants  felt  FanVis  was  a  more 
pleasant  tool  to  use  than  Excel®  (z=1.64,  p=0.1). 

Responses  from  the  retrospective  protocol  varied  for  both  FanVis  and  the  Excel® 
tool.  Participants  were  asked  a  range  of  questions  including  their  impressions  of  the 
tools,  the  aspects  of  the  tools  that  they  liked  the  most,  the  information  they  felt  was 
most  useful  in  completing  their  decision  and  other  related  questions.  The  most  frequent 
response,  in  these  interviews  was  participants'  comments  that  they  were  used  to  Excel®. 
Forty  percent  of  the  participants  made  this  comment.  Overall,  twenty-six  percent  of  the 
participants  felt  that  given  time  they  would  be  able  to  find  data  more  effectively  in 
FanVis.  This  could  be  due  to  the  fact  that  participants  felt  the  views  in  FanVis  conveyed 
more  information,  a  comment  made  by  twenty-six  percent  of  the  participants.  Thirteen 
percent  of  the  participants  commented  that  FanVis  was  more  intuitive,  thirteen  percent 
commented  they  enjoyed  being  able  to  dig  deeper  into  the  data  in  FanVis,  and  thirteen 
percent  commented  FanVis  was  more  visually  appealing.  On  the  other  hand  twenty 
percent  of  the  participants  commented  it  was  easier  for  them  to  view  the  bar  charts  in 
Excel®  and  determine  how  the  requirements  were  being  met  versus  using  FanVis. 

In  terms  of  the  decision  support  tool  designs,  participants  gave  praise  and 
critiques  for  both  FanVis  and  Excel®.  Thirty  percent  of  the  participants  commented  that 
the  center  red  line  in  FanVis,  which  represents  a  baseline  requirements  constraint 
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helped  them  determine  the  baseline  in  FanVis,  and  would  have  liked  to  see  a  similar 
line  in  Excel®.  Thirty-three  percent  of  the  participants  commented  they  were  not  able  to 
discern  the  "-ility"  view  in  FanVis,  thirty-three  percent  commented  they  had  difficulty 
determining  total  cost,  and  twenty-six  percent  commented  they  had  difficulty 
integrating  the  sub-functional  requirement  cost. 

In  addition,  thirteen  percent  of  the  participants  did  not  like  how  "cost"  and  "the 
degree  functional  requirements  were  met"  could  not  be  decoupled.  In  FanVis,  data 
elements  contained  information  regarding  multiple  trade  space  variables  (the  fans) 
encoded  both  cost  information  and  the  degree  the  functional  requirement  was  being 
met.  This  coupling  allowed  participants  to  quickly  gather  information,  but  the 
participants  who  commented  on  this  coupling  would  have  also  liked  to  see  this 
information  presented  separately. 

In  terms  of  other  FanVis  usability  issues,  sixteen  percent  of  the  participants  had 
difficulty  connecting  the  system  legend  in  the  cost  scale  with  the  displayed  systems  in 
the  comparison  view.  Further,  twenty-three  percent  had  difficulty  navigating  within 
FanVis.  These  participants  commented  they  were  uncertain  as  to  which  tab  contained 
the  information  necessary  to  answer  each  question.  Of  the  views  in  FanVis,  participants 
liked  to  use  the  Multi-System  View  most,  as  it  "provided  an  overview  of  all 
information."  They  liked  Comparison  View  best  next,  and  the  Functional  Requirement 
View  the  least. 

Participants  spent  less  time  analyzing  Excel®  because,  as  one  participant  stated, 
"it  is  ordinary."  However,  participants  commented  that  they  did  not  like  the  line  charts 
in  Excel®,  especially  the  cost  categories  tabs.  Forty  percent  of  the  participants 
commented  they  liked  using  the  bar  charts,  while  only  thirteen  percent  commented 
they  liked  utilizing  the  data  tab,  as  it  allowed  them  to  access  the  functional  requirement 
total  cost.  Sixteen  percent  of  the  participants  commented  that  they  would  have  liked  to 
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have  the  information  presented  on  the  Requirements  and  "-ilities"  Tab  and  the  Total 
Cost  Tab  combined. 

Overall,  participants  felt  both  tools  allowed  them  to  complete  the  necessary 
tasks,  and  many  commented  either  tool  was  a  vast  improvement  on  how  they  currently 
complete  acquisitions  (they  mentioned  currently  sifting  through  reams  of  paper). 
Twenty-three  percent  of  the  participants  commented  they  would  have  liked  to  have  the 
total  functional  requirement  cost  in  both  the  tools.  This  metric  was  not  shown  for  the 
purposes  of  this  test  to  determine  how  well  the  participants  could  integrate  information 
with  both  of  the  tools.  Thirteen  percent  felt  automation  could  have  answered  the  system 
acquisition  trade  space  questions  better  than  they  could,  as  absolute  differences  were 
being  judged  in  many  instances.  Additionally,  participants  would  have  liked  to  see 
more  interactivity  within  the  tools.  In  FanVis,  this  could  include  the  ability  to  switch 
from  the  Multi- System  View  to  the  System  View  by  double  clicking  on  one  of  the 
systems.  In  Excel®,  this  interactivity  could  include  the  ability  to  query  for  actual  values, 
or  highlight  specific  system  data. 

5.4.  Cognitive  Strategies 

Analyzing  the  fixation  pattern  of  the  participants  between  the  two  tools  can  yield 
insight  into  the  participant's  cognitive  strategies  [41,  42],  This  section  describes  some 
participants'  fixation  patterns  that  were  common  within  the  two  tools,  and  postulates 
the  similarities  and  differences  caused  by  the  two  different  tools  among  these  pattern 
types,  as  well  as  their  implications. 

There  was  a  considerable  amount  of  noise  present  in  the  eye  tracker  data.  It  is 
possible  that  the  structure  of  the  van  interfered  with  the  signal  from  the  magnetic  sink 
during  all  or  portions  of  the  experiments.  Additionally,  though  participants  had  free 
range  of  head  motion,  the  software  was  not  robust  enough  to  support  changes  in 
posture,  as  a  posture  change  affected  the  overall  position  of  the  participant  relative  to 
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the  screens  and  the  magnetic  source.  Thus,  if  participants  moved  too  much,  their  eye 
track  was  lost.  For  this  reason,  only  five  participants  (3,  6,  16,  23,  and  24)  had  eye  tracks 
which  were  continuously  accurate  for  the  duration  of  both  the  FanVis  and  Excel®  test 
sessions.  Even  for  these  participants,  there  were  instances  of  data  noise.  In  these 
instances,  only  the  overall  pattern  was  analyzed. 

5.4.1.  Patterns  in  FanVis 

In  FanVis,  participants  primarily  demonstrated  fixation  patterns  which  allowed 
them  to  obtain  information  in  the  fastest  and  easiest  manner  possible.  For  instance, 
when  viewing  the  Comparison  View,  the  System  View  or  the  Multi-System  View, 
participants  utilized  near-circular  fixation  patterns.  In  these  three  tabs,  the  data 
elements  are  arranged  around  a  polygonal  structure.  The  near-circular  fixations 
patterns  are  therefore  quite  intuitive.  This  type  of  pattern  allowed  the  participants  to 
scan  from  one  element  to  the  next  in  the  shortest  distance.  Additionally,  participants 
could  maintain  a  continuous  scan,  repeating  around  the  polygon  until  all  necessary 
information  was  gathered.  Further,  the  close  proximity  of  the  labels  to  the  data  elements 
aided  the  manner  which  participants  gathered  information.  For  those  tasks  where  labels 
were  required,  primarily  those  questions  regarding  individual  functional  requirements 
or  "-ilities",  the  participants  did  not  have  to  modify  their  fixation  patterns  within  a 
given  view  but  could  quickly  and  easily  obtain  the  necessary  information  in  the  same 
manner  for  all  task  types. 

For  instance.  Figure  24  depicts  one  of  participant  6's  fixation  patterns  while 
answering  the  question  "Which  system  meets  the  functional  requirements  to  the 
greatest  degree?"  In  the  figure,  the  thin  line  designates  the  participant's  fixation  track. 
The  circles  represent  the  participant's  fixation  location,  with  the  duration  of  this  fixation 
indicated  by  the  size  of  the  circle.  The  bold  arrows  indicate  the  direction  of  these 
fixation  patterns.  In  this  example,  more  than  50  individual  fixations  are  displayed.  It  can 
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be  seen  that  the  participant  moved  from  the  tree  structure  to  the  ID  Objects  fan,  up 
through  the  Track  and  Land  fans,  to  the  tabs  and  the  Launch  fan  and  then  back  down 
between  the  Autonomy  and  Cruise  fans. 


Figure  24:  Participant  6's  fixation  pattern  within  the  System  View 
Typically,  participants  fixated  upon  each  of  the  elements  within  the  "-ility"  View 
from  side  to  side  in  a  horizontal  pattern.  This  fixation  pattern  again  allowed  the 
participants  to  travel  the  shortest  distance  from  one  data  element  to  the  next.  For 
instance.  Figure  25  shows  that  participant  16  first  fixated  between  School  A  and  School 
B  Adaptability,  moved  on  towards  School  C  Adaptability,  where  the  fixation  then 
shifted  down  to  School  C  Reliability  through  School  B  Reliability  to  School  A  Reliability. 
At  this  point  the  participant  either  went  directly  back  up  to  the  views  to  select  the  next 
view  or  through  the  school  labels. 
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Figure  25:  Participant  16's  fixation  pattern  within  the  "-ility"  View 

Though  this  gaze  pattern  looks  different  from  that  in  the  other  views  within  FanVis, 
it  is  based  upon  the  same  principle.  For  the  most  part,  participants  adopted  a  scanning 
pattern  that  allowed  them  to  view  data  elements  in  a  continuous  manner  where  the 
shortest  path  is  taken  from  data  element  to  data  element.  This  type  of  gaze  pattern  is 
efficient  and  simple.  Hence  for  the  "-ility"  view  this  is  a  horizontal  pattern,  and  for  the 
Comparison,  System  and  Multi-System  Views  this  is  a  near-circular  pattern. 

5.4.2.  Patterns  in  Excel® 

In  Excel®,  participants  also  utilized  scanning  patterns  which  allowed  them  to  obtain 
information  in  the  fastest  and  easiest  manner  possible.  However,  because  the  data  labels 
were  placed  further  from  the  data  elements  they  describe,  the  scanning  pattern  varied 
dependent  upon  the  task  type.  For  tasks  which  did  not  require  the  labels,  primarily 
those  questions  on  how  the  system  generally  met  the  system  acquisition  metrics,  the 
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participants  generally  had  a  horizontal  fixation  pattern.  Figure  26  depicts  participant 
23' s  eye  fixation  track  while  answering  the  question,  "Which  system  is  most  balanced  in 
terms  of  meeting  the  "-ilities"?"  Here  the  participant  focused  on  the  "-ility"  legend  seen 
in  the  upper  right  corner  of  the  left  graph.  He  or  she  then  fixated  on  each  of  the  three  “ - 
ilities"  in  turn  going  from  left  to  right  much  in  the  same  way  one  would  read  a 
sentence.  The  far  right  fixation  is  likely  a  transient  fixation  when  the  participant  shifted 
his  or  her  gaze  back  to  the  questions  on  the  right  screen  in  order  to  correctly  identify  the 
answer  as  System  A. 


Figure  26:  Participant  23's  fixation  pattern  within  the  Requirements  and  "-ilities"  Tab 
When  obtaining  information  from  the  data  element  labels  was  required, 
participants  exhibited  a  box  fixation  pattern.  As  shown  in  Figure  27,  this  box  pattern 
allowed  the  participant  to  view  the  labels  of  the  functional  requirements;  followed 
directly  by  the  degree  each  requirement  was  met.  In  this  example,  participant  16 
answered  the  question  "Which  of  the  following  functional  requirements  is  overall  met 
to  the  least  degree  while  maintaining  the  highest  cost?"  This  question  required  the 
participant  to  understand  the  name  of  the  functional  requirements  along  with  the 
degree  they  were  being  met.  In  the  previous  example,  participant  23  only  had  to 
analyze  the  how  the  overall  "-ilities"  were  being  met,  as  compared  to  determining  how 
individual  "-ilities"  were  met.  The  box  pattern  was  therefore  unnecessary.  The 
horizontal  fixation  pattern  is  a  faster  pattern,  as  the  participant's  gaze  does  not  travel  as 
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far  within  the  figure.  Thus,  the  box  fixation  pattern  was  only  used  when  the  participant 
needed  to  know  the  names  of  the  functional  requirements  or  "-ilities."  This  only 
occurred  when  information  pertaining  to  individual  functional  requirements  or 
ilities"  was  necessary. 


Figure  27:  Participant  16's  fixation  pattern  within  the  Requirements  and  "-ilities"  Tab 


5.4.3.  Comparison  of  Eye  Fixation  Patterns 

In  both  FanVis  and  Excel®,  participants  elected  to  use  eye  fixation  patterns  which 
would  allow  them  to  obtain  information  in  the  fastest  and  easiest  way  possible.  For  data 
elements  arranged  in  a  row,  this  typically  meant  a  horizontal  fixation  pattern  which 
would  begin  at  the  right,  go  towards  the  left,  and  on  most  occasions  repeat  back  to  the 
left.  For  data  elements  arranged  in  a  near-circular  fashion,  the  eye  fixation  pattern 
typically  was  near-circular  matching  the  pattern  of  the  elements.  This  pattern  was 
exhibited  for  all  five  participants  on  questions  of  all  three  levels  of  reasoning  difficulty. 
More  importantly,  participants  were  able  to  use  the  same  fixation  pattern  within  a 
single  view  for  all  types  of  tasks  while  using  FanVis,  as  the  data  labels  were  placed  in 
close  proximity  to  the  data  elements  they  were  describing.  When  these  data  label  were 
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necessary  while  using  Excel®,  participants  were  required  to  modify  their  fixation 
patterns.  In  these  instances,  the  participants  utilized  less  efficient  fixation  patterns  as  the 
distances  between  fixations  were  larger.  This  shift  in  the  participants'  fixation  patterns 
could  indicate  the  participants  also  had  to  change  their  cognitive  strategy. 

In  some  instances  the  fixation  patterns  became  more  erratic.  These  erratic  fixations 
were  observed  most  frequently  on  questions  the  participants  did  not  answer  correctly. 
While  erratic  patterns  have  been  shown  as  an  indication  of  poorly  designed  interfaces 
[42],  non-erratic  patterns  were  exhibited  by  the  same  participants  on  questions 
answered  correctly.  From  this  observation,  it  can  be  concluded  that  the  participant's 
cognitive  strategy  changed  between  the  instances  where  the  participant  displayed 
normal  eye  fixation  patterns  and  erratic  patterns.  However,  it  cannot  conclusively  be 
stated  if  a  participant's  uncertainty  on  the  question  caused  him  or  her  to  have  more 
erratic  eye  fixations  or  if  a  lack  of  focused  fixations  caused  the  participant  to  answer  the 
question  incorrectly.  This  is  a  complex  relationship,  which  could  be  analyzed  further  in 
future  work.  Two  examples  are  illustrated  to  highlight  these  eye  fixation  changes. 

One  of  the  most  difficult  questions  for  participants  to  answer  was  question  5, 
"Which  system  is  most  balanced  overall?"  Only  9  participants  out  of  30  responded 
correctly  using  either  of  the  two  tools,  and  no  participant  responded  correctly  using 
both  tools.  Participant  6  responded  incorrectly  to  this  question  with  both  tools, 
spending  31.29  seconds  in  Excel®  and  25.20  seconds  in  FanVis  to  try  to  answer  the 
question.  What  is  interesting  to  note  is  participant  6's  eye  fixation  patterns  in  both 
FanVis  and  Excel®  are  much  more  erratic  than  those  seen  in  his  or  her  eye  fixation 
patterns  on  questions  that  were  correctly  answered.  As  shown  in  Figure  28,  participant 
6  has  no  readily  apparent  track  in  either  FanVis  or  Excel®.  This  same  lack  of  pattern  can 
be  seen  in  other  participants  on  questions  that  were  answered  incorrectly. 
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Figure  28:  Participant  6's  fixation  pattern  within  FanVis(left)  and  Excel®  (right)  on  question  5 
A  second  example  examines  the  fixation  patterns  of  participant  23  while 
answering  the  question  "Which  system  is  most  balanced  in  terms  of  meeting  the 
functional  requirements?"  (Figure  29).  In  FanVis  this  question  was  answered  correctly 
in  42.25  seconds,  while  in  Excel®  it  was  answered  incorrectly  after  15.03  seconds.  In 
FanVis,  a  normal  pattern  is  exhibited,  whereas  in  Excel®  the  participant  has  an  erratic 
eye  pattern. 


Figure  29:  Participant  23's  fixation  pattern  in  FanVis  (right)  and  Excel®  (left)  on  question  11 
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5.4.4.  Implications  of  Eye  Fixation  Patterns 

It  has  been  found  that  humans  generally  exhibit  fixation  patterns  which  are 
tightly  linked  to  the  task.  These  patterns  are  developed  over  time,  and  fall  on  elements 
which  help  promote  the  completion  of  the  task  [32],  This  suggests  that  users  of  any 
display  develop  a  fixation  pattern  most  natural  to  interacting  with  that  display.  Natural 
fixation  patterns  consist  of  small  incremental  movements  between  necessary  elements 
and  is  the  quickest  scanning  pattern  that  could  exist  for  that  task  [32],  Natural  fixation 
patterns  were  exhibited  by  participants  in  both  FanVis  and  Excel®  for  most  of  the 
experiment.  However,  participants  generally  had  more  erratic  fixation  patterns  on 
questions  that  were  answered  incorrectly,  potentially  indicating  the  participant  did  not 
understand  how  to  access  the  data  required  to  answer  that  question. 

To  determine  recommendations  for  a  display  design  so  that  the  quickest  gaze 
pattern  can  be  obtained,  Fitts'  law  should  be  considered.  Fitts'  law  is  a  model  of  human 
movement  and  is  used  to  predict  the  time  it  would  take  a  user  to  point  to  an  area  as  a 
function  of  distance  to  target  and  target  size  [43].  It  has  been  shown  that  Fitts'  law  holds 
for  selection  tasks  using  an  eye  tracking  device  [44],  Though  many  function  variations 
have  been  established,  the  main  relationship  is  given  by  the  equation  Time  log2(D/5) 
where  D  is  the  distance  to  the  target  and  S  is  the  target  size.  Hence,  large  targets  placed 
close  to  each  other  could  be  accessed  faster  than  smaller  targets  spaced  further  apart. 
Thus,  relevant  data  elements  should  be  placed  in  close  proximity  to  foster  faster 
acquisition  of  information.  In  addition,  data  elements  and  their  labels  should  be  placed 
close  to  each  other  if  users  need  to  correlate  names  with  data  elements.  As  was  shown 
in  this  research,  such  proximity  could  allow  users  to  utilize  a  single  fixation  pattern 
regardless  of  the  task.  Participants  had  to  shift  from  a  natural  fixation  pattern  to  a  less 
efficient  pattern  for  a  number  of  tasks  while  using  Excel®.  Though  system  acquisition 
decisions  are  not  time-sensitive  tasks,  it  is  likely  that  the  display  which  allows  users  to 
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utilize  the  most  natural  and  therefore  efficient  pattern  could  also  improve  the  users' 
performance  [42],  Additionally,  a  display  which  promotes  natural  fixation  patterns 
regardless  of  task  could  also  allow  users  to  maintain  a  single  type  of  cognitive  strategy. 

5.5.  Element  Analysis 

To  further  analyze  how  participants  interacted  with  the  data  elements,  it  is 
important  to  determine  what  time  was  spent  fixating  on  elements  relevant  to  the  system 
acquisition  trade  space  questions.  Overall,  participant  spent  an  average  of  66.18%  of  the 
time  fixating  on  relevant  elements  in  FanVis  compared  to  66.58%  in  Excel®.  Hence, 
participants  spent  essentially  the  same  amount  of  time  spent  fixating  on  relevant 
elements  between  the  two  tools.  However,  analyzing  the  percent  time  fixating  on 
relevant  elements  by  Reasoning  Difficulty  Level  yields  some  interesting  trends.  As 
shown  in  Table  3,  participants  have  a  statistically  significant  higher  percentage  of  time 
fixating  on  relevant  elements  in  FanVis  for  Knowledge  Synthesis  questions  compared  to 
Excel®  using  a  Wilcoxon- Signed  Rank  Test  (z=-2.023,  p=0.043).  This  trend  is  reversed  for 
Data  Processing  questions  where  participants  have  a  slightly  higher  percentage  time 
fixating  on  relevant  elements  in  Excel®  compared  to  FanVis,  though  this  pair  was  not 
statistically  significant  (z=-0.730,  p=0.465).  This  trend  is  very  similar  to  the  performance 
trends.  Overall,  one  would  want  to  observe  a  higher  percentage  of  time  spent  fixating 
on  relevant  elements,  as  this  would  indicate  the  participants  spent  a  greater  amount  of 
time  fixating  on  data  elements  useful  to  answering  the  question  at  hand  [45]. 


Table  3:  Average  percent  time  fixating  on  relevant  elements 


FanVis 

Excel® 

Significance 

Knowledge 

75.81% 

65.80% 

Yes  (p  =  0.043) 

Information 

66.65% 

65.26% 

No  (p  =  0.500) 

Data 

64.75% 

64.59% 

No  (p  =  0.465) 
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5.6.  Summary  of  Experimental  Findings 


Results  from  the  human-performance  experiment  led  to  a  range  of  results.  Each 
dependent  variable  provided  insight  into  how  well  the  human  participants  were  able  to 
interact  with  the  two  decision  support  tools  as  they  completed  a  system  acquisition 
decision.  Results  indicated  that  both  tools  had  strengths  and  weaknesses  in  terms  of 
how  well  they  supported  the  human  participant  as  illustrated  by  Table  4.  These 
strengths  and  weaknesses  can  help  guide  how  future  decision  support  tools  are  built  in 
the  future. 


Table  4:  Summary  of  experimental  findings 


Knowledge 

Synthesis 

Information 

Aggregation 

Data  Processing 

Score 

FanVis 

(p=0.046) 

Indistinguishable 

(p=0.437) 

Excel® 

(p=0.027) 

Time 

Indistinguishable 

(p=0.471) 

Indistinguishable 

(p=0.032) 

Excel® 

(p<0.001) 

%  Time  Fixating  on 
Relevant  Elements 

FanVis 

(p=0.043) 

Indistinguishable 

(p=0.50) 

Indistinguishable 

(p=0.46) 

Cognitive  Strategy 

FanVis 

(no  statistical  test) 

Subjective  Opinion 

FanVis 

(p=0.040) 

For  example,  it  was  observed  that  using  FanVis  over  Excel®  statistically  influenced 
how  accurately  participants  answered  the  system  acquisition  questions.  Participants 
obtained  higher  accuracies  in  FanVis  on  Knowledge  Synthesis  questions  and  higher 
accuracies  with  the  Excel®  tool  on  Data  Processing  questions.  Though  time  was  not  as 
important  as  accuracy,  participants  had  similar  response  times  answering  Knowledge 
Synthesis  and  Information  Aggregation  questions  in  FanVis  and  Excel®,  but  faster 
speeds  in  Excel®  on  Data  Processing  questions.  In  addition,  participants  spent  a  greater 
percentage  of  time  fixating  on  relevant  elements  while  completing  Knowledge 
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Synthesis  questions  in  FanVis.  Results  also  suggest  that  FanVis  better  supported  natural 
gaze  patterns,  thus  FanVis  potentially  allowed  participants  to  maintain  their  cognitive 
strategies.  Possibly  these  more  efficient  patterns  could  lead  to  improved  performance, 
especially  when  users  become  more  familiar  to  using  the  tool.  Finally,  participants 
subjectively  believed  FanVis  was  a  better  tool. 

Chapter  6  will  discuss  the  implications  these  results  have  on  the  initial  research 
question  and  the  design  of  future  related  decision  support  tools. 
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6.  Conclusions  and  Future  Work 


The  goal  of  this  research  was  to  determine  the  type  of  decision  support  system 
which  could  best  aid  system  acquisition  decision  makers  as  they  utilize  multiple  levels 
of  reasoning.  This  research  question  was  addressed  through  the  following  methods: 

•  The  investigation  of  the  scope  of  system  acquisitions  and  current  acquisition 
practices  to  determine  the  design  requirements  for  a  system  acquisition  decision 
support  tool  (Chapter  2). 

•  The  design  of  a  system  acquisition  decision  support  tool  which  encompasses  the 
identified  design  requirements  and  best  supports  an  acquisition  decision,  based 
upon  previous  research  (Chapter  3). 

•  The  use  of  human-performance  experimentation  to  evaluate  the  effectiveness  of 
this  tool  versus  a  more  traditional  tool  which  represents  current  state-of-the-art 
decision  tools  (Chapters  4  &  5) 

This  research  sought  to  determine  if  a  configural  decision  support  tool,  FanVis,  is 
able  to  support  high-level  system  acquisition  decisions  better  than  a  traditional 
separable  decision  support  tool.  Excel®.  The  answer  to  this  question  was  determined 
through  analysis  of  participants'  performance  in  the  experiment  outlined  in  Chapter  4. 
Chapter  4  also  examined  subjective  appeal  to  the  tools,  and  the  participants'  cognitive 
strategies  used  to  complete  their  goals. 

6.1.  Performance 

In  general,  both  decision  support  tools  supported  participants  as  they  answered 
questions  pertaining  to  a  system  acquisition  decision  trade  space  on  all  levels  of 
reasoning  difficulty  tested.  In  terms  of  terms  of  speed.  Excel®  allowed  participants  to 
answer  Data  Processing  questions  more  quickly,  though  the  actual  average 
improvement  was  only  60.8  seconds  (essentially  ten  seconds  per  question).  However,  it 
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is  likely  that  acquisition  decision  makers  value  accuracy  over  speed.  In  this  respect, 
FanVis  allowed  the  participants  to  answer  Knowledge  Synthesis  questions  more 
accurately  than  Excel®.  On  the  other  hand.  Excel®  allowed  participants  to  answer  Data 
Processing  questions  more  accurately  than  FanVis,  while  participants  answered 
Information  Aggregation  questions  similarly  with  the  two  tools. 

While  making  an  acquisition  decision,  decision  makers  will  ultimately  be 
answering  Knowledge  Synthesis  questions.  Thus,  it  appears  that  FanVis  could  be  the 
superior  acquisition  decision  support  tool  in  terms  performance.  As  mentioned,  the 
participants  all  had  experience  using  Excel®,  and  as  several  participants  mentioned,  it 
was  "ordinary"  tool  to  use,  while  none  had  ever  seen  or  used  FanVis. 

By  further  examining  performance  at  the  categorical  question  level,  it  was 
determined  which  decision  support  tools  best  supported  users  as  they  extracted  specific 
system  acquisition  trade  space  variables.  FanVis  best  supported  the  extraction  of  sub¬ 
functional  requirement  cost  and  the  Excel®  tool  best  supported  the  extraction  of  "-ility" 
data.  This  information  can  help  guide  the  development  of  new  system  acquisition 
decision  support  tools. 

6.2.  Subjective  Appeal 

Participants  preferred  to  use  the  configural  decision  support  tool  FanVis  over  the 
traditional  separable  decision  support  tool  Excel®  while  completing  a  system  acquisition 
decision.  This  conclusion  is  supported  by  the  fact  that  statistically,  participants  felt 
FanVis  was  a  more  useful  tool,  gave  them  a  better  understanding  of  the  trade  space, 
and  would  choose  to  use  it  in  their  next  acquisition  decision.  This  trend  was  present 
even  though  participants  may  have  preferred  Excel®  due  to  the  fact  that  all  had 
substantial  experience  using  Excel®  and  had  never  seen  FanVis. 

While  participants  preferred  FanVis,  there  were  portions  of  both  decision 
support  tools  that  participants  did  not  understand  or  did  not  enjoy  using.  In  FanVis, 
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some  participants  mentioned  they  had  difficulty  with  the  "-ility"  View,  integrating  the 
sub-functional  requirement  costs,  and  determining  the  total  cost  of  the  system.  With  the 
Excel®  tool,  participants  would  have  liked  a  "meets  requirements"  reference  line  similar 
to  that  in  FanVis.  They  also  believed  the  Cost  Categories  Tab  was  too  cluttered,  and 
would  have  liked  to  see  the  requirements,  "-ilities,"  and  total  cost  plots  displayed  on  the 
same  tab.  For  both  tools,  participants  would  have  liked  to  see  the  total  functional 
requirement  cost,  and  have  more  interactivity  within  the  tools. 

6.3.  Cognitive  Strategies 

The  participants'  cognitive  strategies  were  analyzed  primarily  from  eye  tracker 
data  collected  during  the  experiment.  As  is  typical  in  eye  tracking  studies,  the  data 
obtained  was  often  noisy,  unreliable  in  some  instances,  and  nonexistent  in  others.  Thus, 
using  the  eye  tracker  for  this  experiment  did  not  yield  a  complete  data  set  for  analysis. 
However  a  baseline  for  the  participants'  cognitive  strategies  was  established  from  the 
five  consistent  and  reliable  tracks  obtained. 

In  terms  of  fixation  patterns,  participants  demonstrated  similar  fixation  pattern 
behaviors  with  both  FanVis  and  the  Excel®  tool.  When  participants  understood  the 
question,  and  how  to  access  the  data  required  to  answer  a  question,  they  primarily 
utilized  a  fixation  pattern  that  allowed  them  to  gather  information  in  the  fastest  and 
easiest  manner.  When  participants  did  not  understand  the  question,  and/or  how  to 
access  the  data  they  required  to  answer  that  question,  the  fixation  patterns  were  much 
more  erratic. 

Additionally,  due  to  the  design  of  the  Excel®  tool,  participants  had  to  change 
their  fixation  pattern  dependent  upon  the  type  of  task  being  completed.  For  those  tasks 
where  data  element  names  were  required,  such  as  questions  pertaining  to  individual 
functional  requirements,  participants  used  a  box  fixation  pattern.  This  pattern  is  a  less 
efficient  pattern  than  the  natural  horizontal  fixation  pattern  used  for  tasks  where  data 
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element  names  were  not  required.  FanVis  allowed  the  participants  to  maintain  more 
natural  fixation  patterns  regardless  of  task,  which  suggests  FanVis  supported  a  more 
organized  cognitive  strategy  as  compared  to  Excel®,  although  more  data  should  be 
collected  to  support  these  observations.  This  preliminary  analysis  indicates  FanVis 
could  promote  better  performance  as  more  efficient  eye  fixation  patterns  have  been 
linked  to  improved  performance  [42], 

The  general  trend  in  the  percentage  of  time  participants  spent  fixating  on 
relevant  elements  when  analyzed  by  Reasoning  Difficulty  Level  matched  that  of  the 
trends  observed  in  performance.  Participants  spent  a  larger  percentage  of  time  fixating 
on  relevant  elements  in  FanVis  while  answering  Knowledge  Synthesis  questions,  while 
they  spent  a  larger  percentage  of  time  in  Excel®  while  answering  Data  Processing 
questions.  This  trend  suggests  that  performance  is  increased  in  both  tools  when  users 
spend  a  larger  potion  of  time  fixating  on  elements  that  are  relevant  to  the  task  at  hand 
compared  to  irrelevant  data  elements  or  non-data  elements. 

6.4.  Design  Recommendations 

Design  recommendations  for  a  future  system  acquisition  decision  support  tool  were 
developed  from  these  findings.  These  design  recommendations  stem  from  the 
performance,  subjective,  and  cognitive  strategy  results  of  the  experiment.  These 
recommendations  include  design  modifications  to  the  current  designs,  the  inclusion  of 
additional  metrics,  the  expansion  of  the  use  of  the  element  of  time,  and  the  modification 
of  the  designs  into  a  hybrid  configural-separable  display. 

6.4.1.  Design  Modifications  to  FanVis 

Design  modifications  should  be  made  if  FanVis  were  utilized  as  an  actual  system 
acquisition  decision  support  tool.  Each  of  these  changes  would  likely  increase  the 
usability  of  the  tool,  thus  increasing  the  user's  understanding  of  the  system  acquisition 
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trade  space.  Within  FanVis,  the  greatest  necessary  design  change  is  the  "-ility"  View  in 
its  entirety.  Participants  did  not  understand  this  view,  and  were  thus  not  able  to 
accurately  understand  the  "-ility"  trade  space  data.  Additionally,  more  view 
connectivity  should  be  enabled.  The  user  should  be  able  to  click  on  a  system  in  the 
Multi  System  View  to  access  that  system  in  the  System  View,  or  click  on  a  functional 
requirement  to  access  it  in  the  Functional  Requirement  View.  Such  interactivity  could 
help  the  user  understand  where  necessary  information  is  located.  Additionally,  the 
manner  in  which  cost  is  displayed  should  be  reconsidered.  Participants  had  difficulty 
integrating  the  sub-functional  requirement  costs  as  well  as  determining  the  total  cost. 
Furthermore,  some  participants  had  difficulty  discerning  systems  from  each  other  in  the 
color  scheme. 

6.4.2.  Inclusion  of  Additional  Metrics 

Some  acquisitions  require  additional  metrics  to  those  currently  presented  within 
the  current  designs.  First,  there  is  often  uncertainty  in  the  data  of  a  candidate  system. 
The  decision  maker  must  determine  to  what  degree  the  data  provided  is  accurate.  For 
instance,  in  the  recent  Air  Force  Tanker  acquisition,  the  acquisition  team  estimated 
Boeing's  engineering  costs,  as  they  felt  Boeing's  figures  did  not  accurately  reflect  what 
these  costs  would  be.  However,  these  estimates  were  later  deemed  unreasonable.  This 
unreasonable  estimation  was  one  of  the  causes  cited  for  why  the  Government 
Accountability  office  upheld  Boeing's  protest  to  the  Air  Force's  acquisition  decision 
[46].  It  is  likely  that  if  the  acquisition  team  understood  the  uncertainty  held  in  this 
metric  to  a  greater  extent,  the  resulting  circumstances  would  have  been  different. 

A  second  metric  used  in  some  current  acquisition  decisions  is  risk.  Though  risk 
may  initially  seem  similar  to  uncertainty,  they  are  quite  different  [47],  Risk  is  generally 
measured  as  the  probability  of  an  event  occurring  multiplied  by  the  impact  of  that  event 
occurring  [48-50].  For  example,  if  the  system  were  a  rocket,  one  risk  to  be  considered 
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would  be  a  valve  not  opening,  perhaps  causing  the  rocket  not  to  fire.  Here  the  impact 
would  be  the  rocket  not  firing  and  the  probability  would  be  found  by  determining  how 
often  the  valve  would  malfunction.  For  an  acquisition,  a  system's  risks  encompass  these 
individual  engineering  risks,  but  could  also  include  risks  from  cost,  schedule,  and 
uncertainty  to  name  a  few  [51].  In  this  manner,  there  can  be  hundreds  of  risk  data 
points,  each  symbolizing  one  of  the  different  types  of  risk. 

A  third  metric  that  could  be  included  in  FanVis  is  the  explicit  cost  of  a  functional 
requirement.  The  total  cost  of  a  system  and  the  cost  of  sub-functional  requirement  were 
both  presented.  Participants  could  integrate  the  sub-functional  requirement  cost  to 
obtain  the  functional  requirement  cost,  but  many  mentioned  they  did  not  believe  they 
integrated  this  information  correctly  and  would  have  liked  the  functional  requirement 
cost  presented. 

6.4.3.  Expansion  of  the  Element  of  Time 

An  important  consideration  of  displaying  the  acquisition  trade  space  is  that  of 
time  or  schedule.  Currently  in  FanVis,  cost  is  separated  by  the  system's  life  cycle 
phases.  Breaking  down  the  cost  or  risk  of  the  system  for  a  proposed  schedule  would  aid 
decision  makers  in  understanding  the  temporal  aspects  of  the  project.  Additionally, 
decision  makers  must  determine  if  the  system  can  be  delivered  on  time.  This  is  a  critical 
consideration  as  there  may  be  a  finite  amount  of  time  when  the  organization  needs  the 
system.  Once  this  time  passes,  the  system  is  no  longer  useful. 

6.4.4.  Hybrid  Configural-Separable  Display 

Two  decision  support  tools  were  built  to  test  if  a  configural  or  a  separable 
decision  support  tool  would  best  support  a  system  acquisition  decision  to  a  greater 
degree.  Both  tools  were  able  to  support  this  type  of  decision,  and  each  was  able  to 
support  aspects  of  the  decision  to  a  varying  degrees.  However,  in  order  to  provide  the 
best  support,  it  might  be  most  beneficial  to  design  a  decision  support  tool  that  is  a 
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hybrid  of  a  configural  and  separable  display.  In  this  manner,  high-level  system 
information  could  be  displayed  in  a  configural  manner  so  that  the  larger  constraints  and 
interactions  can  be  easily  pinpointed.  The  decision  maker  can  then  query  for  lower  level 
information  where  it  is  necessary,  which  would  be  then  shown  as  a  separable  display. 
In  this  manner,  the  decision  maker's  display  would  be  dependent  upon  the  task  he  or 
she  was  performing. 

6.5.  Future  Experiment  Recommendations 

In  addition  to  a  set  of  design  recommendations,  a  set  of  recommendations  for  future 
experiments  is  provided.  The  future  experiment  recommendations  stem  from  the 
difficulties  faced  during  the  experiment.  They  include  modifications  to  information 
presented  in  the  system  acquisition  case  study,  and  how  the  eye  tracking  system  was 
used. 

6.5.1.  System  Acquisition  Case  Study 

The  first  recommendation  would  be  to  obtain  a  more  robust  set  of  system 
acquisition  trade  space  data.  The  case  study  used  in  this  experiment  was  from  the 
AUVSI  UAV  student  competition  in  2007.  The  functional  requirements,  cost  figures, 
and  performance  metrics  for  this  case  study  were  obtained  by  analyzing  the  papers  the 
submitted  by  students.  However,  the  metrics  that  could  not  be  found  in  this  manner 
were  estimated.  While  this  provided  a  sufficient  amount  of  data  for  the  purposes  of  the 
experiment,  participants  may  be  able  to  connect  to  the  case  study  to  a  greater  extent  if 
the  numbers  were  more  realistic.  In  this  experiment,  some  participants  commented  that 
they  did  not  understand  some  of  the  functional  requirements  and  "-ilities"  presented  in 
the  case  studies.  Additionally,  an  experiment  could  be  designed  where  the  participants 
would  conduct  system  trades.  In  this  experiment,  participants  could  analyze  different 
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system  outcomes  if  funding  for  a  proposed  system  was  uncertain.  This  would  be  a 
much  more  realistic  experiment,  as  there  is  often  leeway  in  the  design  of  systems. 

6.5.2.  Eye  Tracker  System 

There  was  little  benefit  from  using  an  eye  tracking  system  in  this  experiment.  While 
the  results  found  were  interesting,  they  were  either  common  knowledge  (organized 
gaze  patterns  promote  performance)  or  mirrored  results  found  through  other  metrics 
(percentage  of  time  participants  spent  fixating  on  relevant  elements  mirrored  the  score 
performance  differential).  If  an  eye  tracking  system  is  used  in  the  future,  three 
suggestions  to  improve  the  quality  of  the  results  include: 

1)  To  not  use  a  head  tracking  system  with  a  magnetic  sink  if  the  experiment  takes 
place  inside  a  small  metallic  enclosure.  Even  though  the  manufacturer  stated  no 
problems  should  be  encountered,  it  is  likely  the  experimental  setup  hampered 
the  ability  to  collect  a  greater  amount  of  reliable  eye  tracker  data. 

2)  To  shorten  the  time  period  of  the  experiment,  or  make  the  experiment 
environment  more  comfortable  so  that  the  participants  does  not  move  to  a  great 
of  a  degree.  Even  though  movements  of  the  head  are  allowed  through  the  means 
of  head-tracking,  the  measurements  may  be  more  precise  if  head  movement  is 
limited. 

3)  To  limit  the  screens  the  participant  needs  to  view  to  one  screen.  By  limiting  the 
number  of  places  the  participant  must  look,  head  movement  can  further  be 
limited. 

6.6.  Future  Work 

The  system  acquisition  community  could  greatly  benefit  from  the  introduction  of 
an  organized  and  interactive  system  acquisition  decision  support  tool.  However,  before 
this  can  be  accomplished,  the  design  and  test  of  a  more  robust  system  acquisition 
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decision  support  tool  needs  to  be  conducted.  The  design  recommendations  for  FanVis 
should  be  further  investigated  and  addressed.  In  order  to  enable  FanVis  to  support  a 
wider  range  of  system  acquisitions,  the  metrics  risk  and  uncertainty  should  be  included, 
and  the  element  of  time  should  be  expanded. 

Additionally,  to  truly  be  useful  to  the  system  acquisition  community,  the 
decision  support  tool  must  be  formally  linked  with  the  current  acquisition  process.  For 
both  government  and  commercial  use,  the  decision  support  tool  could  be  linked  to  e- 
procurement.  E-procurement  allows  organizations  to  communicate  requirements, 
resources,  cost  structures,  and  other  data  vital  to  an  acquisition  decision  through  the 
Internet.  It  also  allows  organizations  to  search  for  buyers  or  sellers  of  systems.  It  has 
been  identified  that  the  key  to  using  this  technology  is  to  not  simply  automate  the 
process,  but  allow  the  decision  maker  to  address  regulations,  as  described  in  the  FAR, 
in  a  more  consistent  and  timely  manner  [52],  In  this  manner,  data  provided  by  the 
candidate  organization  to  the  soliciting  organization  can  be  automatically  filtered  into 
the  tool.  This  linkage  is  critical  for  all  acquisitions,  as  it  will  seamlessly  allow  the 
soliciting  organization  to  acquire  and  review  all  data  necessary  for  the  acquisition 
decision. 

Most  importantly,  the  findings  in  this  study  and  others  need  to  be  acted  upon. 
There  is  no  benefit  knowing  the  acquisition  process  is  broken  if  no  positive  steps  are 
taken  to  improve  the  process.  As  Secretary  Gates  mentioned  in  his  Defense  Budget 
Recommendation  Statement  for  2010,  "There  is  broad  agreement  on  the  need  for 
acquisition  and  contracting  reform  in  the  Department  of  Defense.  There  have  been 
enough  studies.  Enough  hand-wringing.  Enough  rhetoric.  Now  is  the  time  for  action 
[53]." 
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Appendix  A  :  Demographic  Descriptive  Statistics 


Category 

N 

Min 

Max 

Mean 

Std. 

Dev. 

Age  (years) 

30 

29 

74 

52.43 

11.39 

Service  in  Armed  Forces  (years) 

14 

4 

42 

16.21 

13.03 

Acquisition  Experience  (years) 

23 

1 

39 

13.95 

13.1 

High-level  Decision  Experience 
(Y/N) 

30/0 

- 

- 

- 

- 

Excel®  Experience  (Y/N) 

30/0 

- 

- 

- 

- 

Gender  (M/F) 

5/25 

- 

- 
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Appendix  B  :  Consent  to  Participate  Form 


CONSENT  TO  PARTICIPATE  IN 
NON-BIOMEDICAL  RESEARCH 

FanVis:  Visualization  Tool  for  System  Acquisition  Decision  Support 

Government  Personnel 

You  are  asked  to  participate  in  a  research  study  conducted  by  Professor  Mary 
Cummings  Ph.D,  from  the  Aeronautics  and  Astronautics  Department  at  the 
Massachusetts  Institute  of  Technology  (M.I.T.).  The  results  of  this  study  may  be 
published  in  a  student  thesis  or  scientific  journal.  You  were  selected  as  a  possible 
participant  in  this  study  because  the  expected  population  this  research  will  influence  is 
expected  to  contain  men  and  women  between  the  ages  of  18  and  65  with  an  interest  in 
making  high-level  decisions.  You  should  read  the  information  below,  and  ask  questions 
about  anything  you  do  not  understand,  before  deciding  whether  or  not  to  participate. 

•  PARTICIPATION  AND  WITHDRAWAL 

Your  participation  in  this  study  is  completely  voluntary  and  you  are  free  to  choose 
whether  to  be  in  it  or  not.  If  you  choose  to  be  in  this  study,  you  may  subsequently 
withdraw  from  it  at  any  time  without  penalty  or  consequences  of  any  kind.  The 
investigator  may  withdraw  you  from  this  research  if  circumstances  arise  which  warrant 
doing  so. 

•  PURPOSE  OF  THE  STUDY 

This  study  is  designed  to  evaluate  if  FanVis,  a  visualization  tool  for  system  acquisition 
decision  support,  is  an  effective  tool  for  completing  system  acquisition  decisions.  To 
evaluate  the  tool's  effectiveness,  the  decisions  and  conclusions  subjects  make  while 
using  FanVis  will  be  compared  to  the  decisions  and  conclusions  that  they  make  while 
using  a  more  traditional  set  of  decision  tools,  mainly  that  of  bar  charts,  line  graphs,  and 
pie  charts.  Factors  that  will  be  taken  into  consideration  while  measuring  the 
effectiveness  of  the  tool,  will  include  your  overall  acquisition  choices,  and 
comprehension  of  the  information  being  displayed,  which  functionalities  you  used 
during  the  process,  as  well  as  your  fixation  points  on  key  visualization  elements,  visual 
scanning  duration  and  location,  and  finally  your  subjective  appeal  to  the  decision  tools. 
Secondary  goals  of  the  study  are  to  determine  additional  tools  that  the  future  decision 
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makers  could  potentially  require  while  using  FanVis,  and  to  asses  the  general  responses 
to  emergent  features  by  comparing  your  eye  focus  location  and  scan  pattern  to  other 
subjects  in  this  study. 

•  PROCEDURES 

If  you  volunteer  to  participate  in  this  study,  we  would  ask  you  to  do  the  following 
things: 

♦  Be  equipped  with  an  adjustable  head-mounted  eye  tracker.  The  eye  tracker  will 
be  explained  and  calibrated. 

♦  Complete  a  baseline  data  proficiency  test  consisting  of  a  series  of  general 
questions  and  tasks  (estimated  time  20  minutes). 

♦  Complete  two  comparison  tasks.  One  task  will  be  completed  with  FanVis,  and 
the  other  with  an  excel  tool.  Prior  to  the  comparison  task,  a  brief  tutorial  will  be 
administered  explaining  the  tool  and  its  features.  Following  this,  a  case  study 
will  be  given  to  you  and  explained.  This  case  study  will  have  all  the  information 
needed  to  complete  the  comparison  task. 

♦  The  comparison  task  will  ask  you  to  determine  which  system  presented  in  the 
case  study  best  fits  the  specific  scenario  outlined.  Once  you  have  you're  your 
decision  choice,  you  will  be  prompted  to  answer  a  series  of  supporting  questions 
regarding  the  visualizations  you  have  just  seen.  Some  of  these  questions  will  also 
ask  if  your  final  decision  choice  would  change  if  various  portions  of  the  scenario 
of  the  system  data  were  slightly  modified.  In  addition  a  small  set  of  questions 
will  be  asked  regarding  how  well  you  liked  using  the  tool  and  if  it  was 
understandable.  You  will  be  able  to  view  and  respond  to  all  comparison  task 
questions  through  the  tool's  interface.  Finally,  you  will  be  asked  to  review  your 
comparison  decision  and  the  steps  made  in  your  determination  by  completing  a 
review  of  the  decision  steps  along  the  way  using  a  retrospective  verbal  protocol 
tool  (estimated  time  30  min  to  1  hour  for  each  task). 

♦  Testing  will  either  occur  at  MIT  in  room  37-301  if  you  are  on  or  near  the  MIT 
campus,  or  near  your  facility  using  HAL's  Mobile  Advanced  Command  and 
Control  Station  (MACCS). 

♦  Total  time:  approximately  2  hours. 

•  POTENTIAL  RISKS  AND  DISCOMFORTS 

There  are  no  anticipated  physical  or  psychological  risks  in  this  study.  There  is  a  slight 
possibility  that  you  will  have  mild  discomfort  from  the  head-mounted  eye- tracker.  You 
may  adjust  the  eye-tracker  at  any  point  in  the  experiment  if  you  desire.  A  brief 
calibration  will  follow  any  adjustments  to  ensure  proper  tracking  of  your  eye.  You  may 
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also  withdraw  from  the  experiment  at  any  time  for  any  reason  without  penalty  or 
consequences  of  any  kind. 

•  POTENTIAL  BENEFITS 

While  there  is  no  immediate  foreseeable  benefit  to  you  as  a  participant  in  this  study, 
your  efforts  will  provide  critical  insight  into  the  effectiveness  of  this  new  decision 
support  tool.  Conclusions  from  this  study  will  yield  better  interfaces  for  high-level 
decision  makers  who  are  facing  system  acquisition  decisions. 

•  PAYMENT  FOR  PARTICIPATION 

This  is  a  voluntary  study.  You  will  not  receive  payment  for  your  time.  The  highest 
performing  individual  will  be  awarded  an  Ipod  Nano.  You  are  eligible  to  receive  this 
award.  Performance  will  be  based  upon  the  number  of  correct  responses  to  questions 
asked  during  the  experiment.  In  case  two  or  more  individuals  obtain  the  same  number 
of  correct  responses,  the  tie  will  be  resolved  by  awarding  the  prize  to  the  individual 
with  the  fastest  response  time. 

•  CONFIDENTIALITY 

Any  information  that  is  obtained  in  connection  with  this  study  and  that  can  be 
identified  with  you  will  remain  confidential  and  will  be  disclosed  only  with  your 
permission  or  as  required  by  law.  You  will  be  assigned  a  subject  number  which  will  be 
used  on  all  related  documents  to  include  databases,  summaries  of  results,  etc.  Only  one 
master  list  of  subject  names  and  numbers  will  exist  that  will  remain  only  in  the  custody 
of  Professor  Cummings. 

Your  eye  will  be  videotaped  during  this  experiment  for  purposes  of  eye  position  data 
extraction.  By  signing  this  form  you  agree  to  have  your  eye  videotaped  throughout  the 
duration  of  the  experiment.  You  have  the  right  to  review  the  video  tape.  If  you  wish  to 
do  so,  please  notify  the  experimenter.  All  interaction  with  the  computer  system  will  be 
recorded  via  tracking  software  for  reproducibility  purposes.  All  video,  tracking  and 
other  data  will  be  stored  in  electronic  files  under  your  subject  number,  and  will  only 
accessible  by  the  experimenter  and  the  principal  investigator.  At  no  point  will  your 
personal  data  be  released  unless  required  by  law.  At  no  point  will  the  experimental  data 
and  research  analysis  be  released  in  a  manner  that  allows  for  your  identification.  If 
further  use  of  your  experimental  data  is  needed,  you  will  be  contacted  for  consent. 

•  IDENTIFICATION  OF  INVESTIGATORS 
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If  you  have  any  questions  or  concerns  about  the  research,  please  feel  free  to  contact  the 
Principal  Investigator,  Mary  L.  Cummings,  at  (617)  252-1512,  e-mail,  missyc@mit.edu, 
and  her  address  is  77  Massachusetts  Avenue,  Room  33-305,  Cambridge,  MA  02139. 

•  EMERGENCY  CARE  AND  COMPENSATION  FOR  INJURY 

In  the  unlikely  event  of  physical  injury  resulting  from  participation  in  this  research  you 
may  receive  medical  treatment  from  the  M.I.T.  Medical  Department,  including 
emergency  treatment  and  follow-up  care  as  needed.  Your  insurance  carrier  may  be 
billed  for  the  cost  of  such  treatment.  M.I.T.  does  not  provide  any  other  form  of 
compensation  for  injury.  Moreover,  neither  the  offer  to  provide  medical  assistance  nor 
the  actual  provision  of  medical  services  shall  be  construed  as  an  admission  of 
negligence  or  acceptance  of  liability.  Questions  regarding  this  policy  may  be  directed  to 
M.I.T's  Insurance  Office,  (617)  253-2823. 

•  RIGHTS  OF  RESEARCH  SUBJECTS 

You  are  not  waiving  any  legal  claims,  rights  or  remedies  because  of  your  participation 
in  this  research  study.  If  you  feel  you  have  been  treated  unfairly,  or  you  have  questions 
regarding  your  rights  as  a  research  subject,  you  may  contact  the  Chairman  of  the 
Committee  on  the  Use  of  Humans  as  Experimental  Subjects,  M.I.T.,  Room  E25-143B,  77 
Massachusetts  Ave,  Cambridge,  MA  02139,  phone  1-617-253  6787. 
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SIGNATURE  OF  RESEARCH  SUBJECT  OR  LEGAL  REPRESENTATIVE 


I  understand  the  procedures  described  above.  My  questions  have  been  answered  to  my 
satisfaction,  and  I  agree  to  participate  in  this  study.  I  have  been  given  a  copy  of  this 
form. 


Name  of  Subject 


Name  of  Legal  Representative  (if  applicable) 


Signature  of  Subject  or  Legal  Representative 


Date 


SIGNATURE  OF  INVESTIGATOR 


In  my  judgment  the  subject  is  voluntarily  and  knowingly  giving  informed  consent  and 
possesses  the  legal  capacity  to  give  informed  consent  to  participate  in  this  research 
study. 


Signature  of  Investigator 


Date 
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Appendix  C  :  Demographic  Survey 

1.  Age: _ 

2.  Gender:  □  Male  □  Female 

3.  Native  Language: _ 

If  native  language  is  not  English: 

English  Proficiency: 

□  Low 

□  Moderate 

□  High 

4.  Occupation: _ 

If  student: 

a)  Class  Standing:  □  Undergraduate  □  Graduate 

b)  Major: _ 

If  currently  or  formerly  associated  with  any  country’s  armed  forces: 

a)  Country/State: _ 

b)  Status:  □  Active  Duty  □  Reserve  □  Retired  □  DOD  Civilian  □  Other _ 

c)  Sendee:  □  Army  □  Navy  □  Air  Force  □  Other _ 

d)  Rank: _ 

e)  Years  of  Sendee: _ 

5.  Have  you  had  experience  making  high-level  decisions  for  a  team,  project,  or 
organization? 

□  Yes 

□  No 

If  yes,  extent  of  experience: 

□  Low 

□  Medium 

□  High 

6.  Have  you  had  system  acquisition  experience? 

□  Yes 

□  No 

If  yes: 

a)  Type  of  system(s)  acquired: _ 

b)  Reason  systeni(s)  acquired: _ 

c)  Duration  (years  or  hours): _ 
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7.  Have  you  had  experience  using  data  manipulation  tools  (such  as  Microsoft  Excel, 
MATLAB,  Mathematica,  Maple,  Statistical  analysis  software,  etc...)  ? 

□  Yes 

□  No 

If  yes: 

a)  Types  or  names  of  products  used: _ 

b)  Exten  t  of  experience: 

□  Low 

□  Moderate 

□  High 

c)  Exten  t  of  usage  on  a  daily  basis: 

□  Low 

□  Moderate 

□  High 

8.  Are  you  color  blind? 

□  Yes 

□  No 

If  yes: 

Which  type  of  color  blindness  (if  known) _ 
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2)  In  1965,  private  school  expenditures  were  approximately  what  percentage  of  total 
expenditures? 

a)  15% 

b)  20% 

c)  25%  -  correct 

d)  30% 

PUBLIC  AND  PRIVATE  SCHOOL  EXPENDITURES 
1965-1979 
(in  billions  of  dollars) 


3)  Which  of  the  following  categories  has  the  greatest  increase  in  workers  between  1981  and 
1995? 

a)  Blue  Collar 

b)  Service 

c)  Farm 

d)  Professional  -  correct 
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4)  In  which  year  did  the  number  of  applications  increase  the  most  from  the  previous  year? 

a)  1984 

b)  1985 

c)  1986 -correct 

d)  1987 


Number  of  Graduate  Student  Applicants 
at  University  X,  1982-1991 


5)  The  difference  between  the  profits  for  product  A  and  B  is  greatest  in  which  year? 

a)  1996 

b)  1997 -correct 

c)  1998 

d)  1999 

Profits  for  the  AB  Company, 


6)  In  how  many  years  was  the  average  number  of  pages  per  newspaper  at  least  twice  as  much  as 
the  average  in  1940? 

a)  One 

b)  Two 


110 


c)  Three  -  correct 

d)  Four 

AVERAGE  NUMBER  OF  PAGES  PER  NEWSPAPER  FOR  NEWSPAPAER  X 


1940  1950  1960  1970  1980 

YEAR 

7)  How  many  boys  attended  the  1995  convention? 

a)  716 

b)  540 


c)  358  -  correct 

d)  225 


Total  number  in 
attendance 


Year 

Number 

1995 

716 

1996 

1108 

1997 

1520 

1998 

2244 

8)  Which  two  years  did  the  least  number  of  boys  attend  the  convention? 

a)  1995  &  1996  -  correct 

b)  1995  &  1998 

c)  1996  &  1998 

d)  1997  &  1998 
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Percentage  of  Boys  vs.  Girls  in  Attendance  at  a 
National  Youth  Convention 


Total  number  in 
attendance 


Year 

Number 

1995 

716 

1996 

1108 

1997 

1520 

1998 

2244 
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Appendix  E  :  Case  Studies  for  Test  Sessions 


Your  Mission 

••• 

•••• 

••••• 

•  ••• 

•  ••• 
••• 

•  • 

•  You  are  an  executive  at  AMI  Funding,  where 

you 

FanVis  Case  Study 

find  projects  worthy  of  your  company’s  financial 

support 

•  •• 

•  You  would  like  to  invest  in  a  student  group  that  is 

•  •••• 

developing  an  Unmanned  Aerial  Vehicles  (or 

•  ••• 

•  ••• 

UAV  for  short) 

•  •• 

•  Future  commercial  endeavors  would  more  than  likely 

•  • 

profit  the  company 

•  All  the  information  you  will  need  is  already 

loaded  in  a  tool  for  you  to  use,  this  brief  simply 

will  give  you  an  overview 
*^i 

> 

UAV  Overview  I 


The  UAV  need  to  have  three  system 

characteristics,  or  “-ilities” 

•  Adaptability  -  adapt  to  changes  in  the  types  of 
missions  it  performs 

•  Reliability  -  perform  desired  functions  when  requested 
for  a  time  span  of  at  least  3  years 

•  Sustainability  -  have  the  ability  to  be  repaired,  or 
upgraded  as  necessary  during  its  lifespan 


UAV  Overview  II 


In  addition,  they  need  to  fulfill  7  basic  functional 
requirements: 

•  Launch 

•  Transition  to  Autonomy 

•  Maintain  Flight 

•  Navigate  Course 

•  Locate  Objects 

•  Track  Objects 

•  Land 


UAV  Performance 

•  You  have  selected  3  UAVs  named  here  as  School  A, 
School  B  and  School  C 

•  You  have  determined  how  well  each  system  fulfills  the 
system  “-ilities”  and  requirements 

•  This  information  has  been  entered  into  a  tool  for  you  to  evaluate 

•  You  have  created  a  scale  to  represent  how  well  systems 
meet  these  requirements: 


Requirement  Categories: 

Does  not  meet: 

1 

Partially  meets: 

2 

Meets: 

3 

Exceeds: 

4 

Exceeds  Greatly: 

5 

“-ilities''  Categories: 
Does  not  meet 
Meets:  1 

Exceeds:  1 


Sub-functional  Requirements 


The  sub-fiinctional  requirements  allow  the  larger 
functional  requirement  to  be  achieved 
For  example  the  functional  requirement  Land 
requires  the  following  sub-functional 
requirements: 

•  Plane  configured  for  landing 

•  Deceleration 

•  Controlled  descent 

•  Impact  with  runway 

•  Come  to  full  stop  * 
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Cost  of  UAV 

•  You  are  concerned  with  how  much  it  will  cost  to  develop 
the  system 

•  You  also  want  to  make  sure  that  one  functional 
requirement  does  not  dominate  the  total  cost  of  the  UAV 

•  You  have  asked  each  UAV  company  to  report  how  much 
it  costs  to  develop  its  system  broken  down  over  the  sub¬ 
functional  requirements  of  the  system  by  three  cost 
phases  of  its  development 

•  Research  and  Development  (R&D) 

•  Production 

•  Operations 


Cost  breakdown  example 

•  The  systems  took  the  functional  and  associated 
sub-functional  requirements  you  gave  them,  and 
provided  you  with  a  cost  breakdown  such  as  the 
one  below  for  School  A: 


System  Selection  Criteria 

When  making  your  system  decisions  please  use  the 

following  baseline  criteria.  This  will  be  provided  for  you 
on  your  screen  along  with  other  criteria  scenarios  which 
you  may  need  during  your  decision. 

•  Meet  or  exceed  “ilities” 

•  Meet  or  exceed  Functional  Requirements 

•  Minimize  Cost 

•  Balance 

•  “ility”  across  system 

•  Functional  Requirement  across  system 

•  Cost  across  sub-functional  requirement 

•  Maximize  degree  “ilities”  and  requirements  met 


Your  task 

•  Now  that  you  have  all  the  information  you  need  to 
determine  a  system  to  choose 

•  All  the  information  you  need  has  already  been 
loaded  into  the  comparison  tool  named  FanVis 
you  will  be  using 

•  Once  you  have  determined  that  choice  on  the 
questionnaire  panel,  a  series  of  questions  will 
follow  to  solidify  your  answer 
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Your  Mission 


Excel  Case  Study 


•  You  are  head  of  the  Mika  District  Education 
Board,  a  district  with  50%  of  its  population  below 
the  poverty  line 

•  You  would  like  to  improve  the  district  by 
purchasing  hundreds  of  new  low  cost  educational 
laptops 

•  Enable  students  to  explore  and  learn  better  with  their 
teachers  and  peers 

•  All  the  information  you  will  need  is  already 

loaded  in  a  tool  for  you  to  use,  this  brief  simply 
will  give  you  an  overview  , 


Laptop  Overview  I 

•  The  laptops  need  to  have  three  system 

characteristics,  or  “-ilities” 

•  Usability  -  be  easy  to  use  by  a  variety  of  people,  some 
of  which  have  never  used  a  computer  before 

•  Reliability  -  perform  desired  functions  when  requested 
for  a  time  span  of  at  least  3  years 

•  Modularity  -  consist  of  small  subsets  that  can  be 
combined,  replaced,  or  upgraded  when  desired 


Laptop  Overview  II 

•  In  addition,  they  also  need  to  fulfill  7  basic 
functional  requirements: 

•  Support  State  Settings 

•  Secure  Data 

•  GUI  Platf armed 

•  File  Manipulation 

•  Content  Manipulation 

•  Information  Sharing 

•  Modify  Hardware 


Laptop  Performance 


You  have  selected  3  laptops  named  here  as  System  1, 
System  2,  and  System  3 
You  have  determined  how  well  each  system  fulfills  the 
system  “-ilities”  and  requirements 

•  This  information  has  been  entered  into  a  tool  for  you  to  evaluate 

You  have  created  a  scale  to  represent  how  well  systems 
meet  these  requirements: 


Requirement  Categories: 

Does  not  meet: 

-I 

“-ilities"  Categories: 

Partially  meets: 

2 

Does  not  meet:  1 

Meets: 

3 

Meets:  2 

Exceeds:  3 

Exceeds  Greatly: 

5 

Sub-functional  Requirements 


The  sub-functional  requirements  allow  the  larger 
functional  requirement  to  be  achieved 
For  example  Secure  Data  requires  the  following 
sub-functional  requirements: 

•  Log-in  protection 

•  Support  Updates 

•  Embedded  Software 

•  Automatic  Backups 

•  Data  Recovery 


Cost  of  Laptop 


Beyond  how  well  the  systems  meet  requirements,  you  are 
concerned  with  how  much  all  these  laptops  will  cost 
You  want  to  make  certain  that  neither  a  single  sub¬ 
functional  nor  a  functional  requirement  dominate  the  total 
cost  of  the  laptop 

You  have  asked  each  laptop  company  to  report  how  much 
it  costs  to  develop  its  system  broken  down  over  the  sub- 
functional  requirements  of  the  system  by  three  cost 
phases  of  its  development 

•  Production 

•  Testing 

•  Maintenance 


Cost  breakdown  example 


The  systems  took  the  functional  and  associated  sub¬ 
functional  requirements  you  gave  them,  and  provided  you 
with  a  cost  breakdown  such  as  the  one  below  for  System  1 : 
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System  Selection  Criteria 

When  making  your  system  decisions  please  use  the 
following  baseline  criteria.  This  will  be  provided  for  you 
on  your  screen  along  with  other  criteria  scenarios  which 
you  may  need  during  your  decision. 

•  Meet  or  exceed  “ilities” 

•  Meet  or  exceed  Functional  Requirements 

•  Minimize  Cost 

•  Balance 

•  “ility”  across  system 

•  Functional  Requirement  across  system 

•  Cost  across  sub-functional  requirement 

•  Maximize  degree  “ilities”  and  requirements  met 


Your  task 

•  Now  that  you  have  all  the  information  you  need  to 
determine  a  system  to  choose 

•  All  the  information  you  need  has  already  been 
loaded  into  the  an  Excel®  spreadsheet  you  will  be 
using 

•  Once  you  have  determined  that  choice  on  the 
questionnaire  panel,  a  series  of  questions  will 
follow  to  solidify  your  answer 
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Appendix  F  :  Tutorials  for  Test  Sessions 


FanVis  Tutorial 

FanVis  Tutorial 

•  FanVis  tool  is  a  new  decision  support  visualization  to 
intended  to  be  used  while  making  system  acquisition 

•  •• 

•  ••• 

■  •••• 

•  ••• 

•  ••• 

•  •• 

•  • 

ol 

_ 

•  •• 

•  ••• 

•  •••• 

•  ••• 

•  ••• 

•  •• 

•  • 

•  As  you  are  going  through  the  tutorial,  interact  with  the 
interface,  try  to  answer  the  questions  in  these  slides  and 
ask  any  questions  you  might  have 

•  The  interface  has  a  modified  case  study  for  you  to  use. 

You  will  get  a  different  set  of  data  for  your  decision  task. 
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Functional  Requirement  View 


The  functional  requirement  view  gives 
you  a  detailed  look  at  one  of  the 
functional  requirements. 

The  percentage  of  each  fen  is  shaded 
indicates  the  sub-functional 
requirement  cost  compared  to  most 
expensive  sub-functional  requirement 
shown  in  this  view. 

Which  system  looks  like  they  spent 
the  least  on  this  functional 
requirement?  The  most? 


These  systems  are 
exceeding  this 
requirement  -  they  ai 
above  the  red  line 


5T 


This  system  is  not 
meeting  this 
requirement  -  it  is 
below  the  red  line 


End  of  Tutorial 

•  You  have  now  reached  the  end  of  the  tutorial 


Y ou  may  take  some  time  now  to  review  any 
portion  of  this  tutorial  if  you  wish.  Once  you  are 
done  your  decision  process  will  begin. 

Again,  feel  free  to  ask  any  questions  and  good 
luck  with  your  decision  making  process! 
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•  •• 

•  ••• 

Excel  Spreadsheet 

Excel  Spreadsheet  Tutorial 

•  ••• 

•  •• 

•  • 

Tutorial 

•  This  tutorial  will  show  you  the  different  features  that  are 

included  in  the  Excel  spreadsheet  you  will  be  using 

•  •• 

•  ••• 

•  As  you  are  going  through  the  tutorial,  interact  with  the 

•  ••• 

interface,  try  to  answer  the  questions  in  these  slides  and 

•  ••• 

•  •• 

ask  any  questions  you  might  have 

•  • 

•  The  interface  has  a  modified  case  study  for  you  to  use. 

You  will  get  a  different  set  of  data  for  your  decision  task. 

11 

_ 

12 

Spreadsheet  Anatomy 


There  are  four  different  sheets  in  the  spreadsheet;  Data, 
Requirements  and  “-ilities”,  Total  Cost  and  Cost 
Categories 

Requirements  and  “-ilities”,  total  Cost  and  Cost 
Categories  each  shows  you  a  different  set  of  charts 
You  can  select  one  of  the  sheets  to  view  by  clicking  on 
its  tab  at  the  bottom  left  corner. 
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49 

50 

Mil  ►!  \  Data/  Requirements  and  Ilities  /  Total  Cost  \Cost  Categories/ 

Requirements  and  ‘‘-ilities’' 


The  Requirements  and  '‘-ilities” 
page  shows  how  well  the 
systems  are  performing  on 
meeting  “-ilities”  and  functional 
requirements 

The  larger  the  bar  is,  the  more 
that  requirement  is  being  met. 

Is  there  a  system  that  is  not 
meeting  all  the  “-ilities”  ? 

Is  there  a  system  that  is  not 
meeting  all  the  requirements? 
Which  system  is  most  balanced 
on  the  “-ilities”?  Functional 
Requirements? 


Data  Sheet  Overview 

•  You  will  not  need  to  change  any  information  in  the  data 
sheet 

•  The  data  sheet  comprises  of  two  sections  in  case  you 
would  like  to  reference  this  sheet: 

•  System  Information  Section  -  Add  or  remove  systems,  functional 
requirements,  sub-functional  requirements,  “-ilities”,  and  cost 
categories 

•  System  Data  Section  -  Change  the  degree  to  which  systems  meet 
functional  requirements  and  “-ilities”,  and  change  how  much 
each  sub-functional  requirement  costs 


End  of  Tutorial 

•  You  have  now  reached  the  end  of  the  tutorial 

•  You  may  take  some  time  now  to  review  any 
portion  of  this  tutorial  if  you  wish.  Once  you  are 
done  your  decision  process  will  begin. 

•  Again,  feel  free  to  ask  any  questions  and  good 
luck  with  your  decision  making  process! 
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Appendix  G  :  System  Acquisition  Trade  Space  Questions 

Questions  while  testing  within  FanVis: 

Knowledge  Synthesis  Questions 

1)  Which  system  best  meets  the  baseline  system  selection  criteria? 

a)  School  A  -  correct 

b)  School  B 

c)  School  C 

d)  Uncertain 

2)  What  system  best  meets  the  system  selection  criteria  #2?  This  criteria  asks  to 
maximize  how  the  functional  requirements  Cruise  and  Navigate  are  met  before 
completing  the  baseline  criteria. 

a)  Schools  A 

b)  School  B 

c)  School  C  -  correct 

d)  Uncertain 

3)  Which  of  the  following  functional  requirements  for  School  C  best  meets  the  following 
set  of  criteria:  Maximize  degree  met,  and  minimize  total  cost. 

a)  Launch  -  correct 

b)  Cruise 

c)  Track 

d)  Land 

4)  Which  system  meets  the  functional  requirements  and  the  "-ilities"  to  the  greatest 
degree? 

a)  Schools  A-  correct 

b)  School  B 

c)  School  C 

d)  Uncertain 

5)  Which  system  is  most  balanced  overall?  This  balance  is  in  terms  of  meeting  the 
functional  requirements  and  "-ilities"  as  well  as  total  cost  per  sub-functional 
requirement? 

a)  Schools  A 

b)  School  B-  correct 

c)  School  C 

d)  Uncertain 

6)  Which  of  the  following  functional  requirements  is  overall  met  to  the  least  degree 
while  maintaining  the  highest  cost? 
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a)  Launch 

b)  Cruise 

c)  Track  -  correct 

d)  Land 

Information  Aggregation  Questions 

7)  Which  system  balances  the  cost  per  sub-functional  requirement  to  the  greatest 
degree? 

a)  Schools  A 

b)  School  B-  correct 

c)  School  C 

d)  Uncertain 

8)  Which  system  meets  the  functional  requirements  to  the  greatest  degree? 

a)  Schools  A-  correct 

b)  School  B 

c)  School  C 

d)  Uncertain 

9)  Which  system  is  most  balanced  in  terms  of  meeting  the  "-ilities"? 

a)  Schools  A 

b)  School  B 

c)  School  C-  correct 

d)  Uncertain 

10)  Which  of  the  following  functional  requirements  is  most  expensive  for  School  A? 

a)  Launch 

b)  Cruise 

c)  Track  -  correct 

d)  Land 

11)  Which  system  is  most  balanced  in  terms  of  meeting  the  functional  requirements? 

a)  Schools  A 

b)  School  B-  correct 

c)  School  C 

d)  Uncertain 

12)  Which  system  meets  the  "-ilities"  to  the  greatest  extent? 

a)  Schools  A-  correct 

b)  School  B 

c)  School  C 

d)  Uncertain 
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Data  Processing  Questions 

13)  Which  system  has  the  greatest  total  cost? 

a)  Schools  A 

b)  School  B 

c)  School  C-  correct 

d)  Uncertain 

14)  Which  system  meets  the  functional  requirement  Launch  to  the  greatest  degree? 

a)  Schools  A 

b)  School  B 

c)  School  C-  correct 

d)  Uncertain 

15)  Which  "-ility"  does  School  B  meet  to  the  greatest  extent? 

a)  Adaptability 

b)  Reliability-  correct 

c)  Sustainability 

d)  Uncertain 

16)  The  most  expensive  sub-functional  requirement  for  School  A  is  under  which 
functional  requirement? 

a)  Autonomy 

b)  Navigate 

c)  ID  Objects-  correct 

d)  Land 

17)  Which  of  the  following  functional  requirements  does  School  C  meet  to  the  least 
degree? 

a)  Launch 

b)  Cruise 

c)  ID  Objects 

d)  Track-  correct 

18)  Which  system  meets  the  "-ility"  Sustainability  to  the  greatest  degree? 

a)  Schools  A-  correct 

b)  School  B 

c)  School  C 

d)  Uncertain 

Repeated  Knowledge  Synthesis  Question 

19)  After  answering  these  questions,  which  system  do  you  now  believe  meets  the 
baseline  system  selection  criteria? 

a)  Schools  A-  correct 

b)  School  B 

c)  School  C 

d)  Uncertain 
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Questions  while  testing  within  the  Excel®  Tool: 

Knowledge  Synthesis  Questions 

1)  Which  system  best  meets  the  baseline  system  selection  criteria? 

a)  System  1 

b)  System  2 

c)  System  3-  correct 

d)  Uncertain 

2)  What  system  best  meets  the  system  selection  criteria  #2?  This  criteria  asks  to 
maximize  how  the  functional  requirements  File  Manipulation  and  Content 
Manipulation  are  met  before  completing  the  baseline  criteria. 

a)  System  1-  correct 

b)  System  2 

c)  System  3 

d)  Uncertain 

3)  Which  of  the  following  functional  requirements  for  School  C  best  meets  the  following 
set  of  criteria:  Maximize  degree  met,  and  minimize  total  cost. 

a)  Secure  Data-  correct 

b)  GUI  Plat-formed 

c)  Content  Manipulation 

d)  Information  Sharing 

4)  Which  system  meets  the  functional  requirements  and  the  "-ilities"  to  the  greatest 
degree? 

a)  System  1 

b)  System  2 

c)  System  3-  correct 

d)  Uncertain 

5)  Which  system  is  most  balanced  overall?  This  balance  is  in  terms  of  meeting  the 
functional  requirements  and  "-ilities"  as  well  as  total  cost  per  sub-functional 
requirement? 

a)  System  1 

b)  System  2-  correct 

c)  System  3 

d)  Uncertain 

6)  Which  of  the  following  functional  requirements  is  overall  met  to  the  least  degree 
while  maintaining  the  highest  cost? 

a)  Secure  Data-  correct 

b)  GUI  Plat-formed 

c)  Content  Manipulation 

d)  Information  Sharing 
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Information  Aggregation  Questions 

7)  Which  system  balances  the  cost  per  sub-functional  requirement  to  the  greatest 
degree? 

a)  System  1 

b)  System  2-  correct 

c)  System  3 

d)  Uncertain 

8)  Which  system  meets  the  functional  requirements  to  the  greatest  degree? 

a)  System  1 

b)  System  2 

c)  System  3-  correct 

d)  Uncertain 

9)  Which  system  is  most  balanced  in  terms  of  meeting  the  "-ilities"? 

a)  System  1 

b)  System  2-  correct 

c)  System  3 

d)  Uncertain 

10)  Which  of  the  following  functional  requirements  is  most  expensive  for  System  3? 

a)  Secure  Data 

b)  GUI  Plat-formed-  correct 

c)  Content  Manipulation 

d)  Information  Sharing 

11)  Which  system  is  most  balanced  in  terms  of  meeting  the  functional  requirements? 

a)  System  1 

b)  System  2-  correct 

c)  System  3 

d)  Uncertain 

12)  Which  system  meets  the  "-ilities"  to  the  greatest  extent? 

a)  System  1 

b)  System  2 

c)  System  3-  correct 

d)  Uncertain 
Data  Processing  Questions 

13)  Which  system  has  the  greatest  total  cost? 

a)  System  1-  correct 

b)  System  2 

c)  System  3 

d)  Uncertain 
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14)  Which  system  meets  the  functional  requirement  Modify  Hardware  to  the  greatest 
degree? 

a)  System  1 

b)  System  2 

c)  System  3-  correct 

d)  Uncertain 

15)  Which  "-ility"  does  System  2  meet  to  the  greatest  extent? 

a)  Usability 

b)  Modularity-  correct 

c)  Reliability 

d)  Uncertain 

16)  The  most  expensive  sub-functional  requirement  for  System  3  is  under  which 
functional  requirement? 

a)  Support  State  Setting-  correct 

b)  File  Manipulation 

c)  Information  Sharing 

d)  Modify  Hardware 

17)  Which  of  the  following  functional  requirements  does  School  C  meet  to  the  least 
degree? 

a)  Support  State  Setting 

b)  Secure  Data 

c)  GUI  Plat-formed-  correct 

d)  Content  Manipulation 

18)  Which  system  meets  the  "-ility"  Reliability  to  the  greatest  degree? 

a)  System  1 

b)  System  2 

c)  System  3-  correct 

d)  Uncertain 

Repeated  Knowledge  Synthesis  Question 

After  answering  these  questions,  which  system  do  you  now  believe  meets  the  baseline 
system  selection  criteria? 

a)  System  1 

b)  System  2 

c)  System  3-  correct 

d)  Uncertain 
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Appendix  H  :  Retrospective  Protocol 

1)  What  was  your  first  impression  of  the  tools? 


2)  What  was  your  impression  of  the  systems  with  regards  to  the  tools? 


3)  Was  there  information  that  you  felt  you  needed  and  couldn’t  find,  or  had  difficulty  finding? 


4)  What  information  helped  you  determine  which  system  was  best  in  each  tool? 


5)  Were  there  aspects  of  the  tools  that  you  felt  was  distracting? 


6)  Was  there  a  portion  of  the  tool  (a  graph  or  a  page)  that  you  liked  the  most?  If  so  what  was  it? 


7)  Was  there  a  portion  of  the  tool  (a  graph  or  a  page)  that  you  liked  the  least?  If  so  what  was  it? 
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Appendix  I :  Preferred  Tool  Selection  Questionnaire 


I  felt  that _ was  more  useful  as  a  decision  support  tool. 

□  FanVis  □  Excel® 

I  felt  that _ was  a  more  pleasant  tool  to  use. 

□  FanVis  □  Excel® 

Overall  I  felt _ left  me  with  a  better  understanding  of  the  system  acquisition  trade  space. 

□  FanVis  □  Excel® 

Given  the  opportunity,  I  would  choose _ to  complete  my  next  system  acquisition  decision. 

□  FanVis  □  Excel® 

Please  provide  any  last  thoughts  or  comments  on  either  the  tools,  or  the  experiment  itself. 
Thank  you! 
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Appendix  J  :  Statistics  for  Time 


Solution  for  Fixed  Effects 

Effect 

Decision 

Reason 

Estimate 

Std. 

Error 

DF 

t  value 

Pr>  1 1 1 

Intercept 

5.7632 

0.1367 

28 

42.14 

<.0001 

Proficiency  Time 

0.00204 

0.00043 

56 

4.67 

<.0001 

Decision  Support 

Tool 

Excel® 

0.0493 

0.0681 

29 

0.73 

0.4741 

Decision  Support 

Tool 

Fan  Vis 

Reasoning  Difficulty 
Level 

Data 

Processing 

-1.0198 

0.0759 

58 

-13.43 

<.0001 

Reasoning  Difficulty 
Level 

Information 

Aggregation 

-0.7816 

0.0759 

58 

-10.30 

<.0001 

Reasoning  Difficulty 
Level 

Knowledge 

Synthesis 

Decision*Reasoning 

Excel® 

Data 

Processing 

-0.3954 

0.0920 

56 

-4.28 

<.0001 

Decision*Reasoning 

Excel® 

Information 

Aggregation 

-0.1974 

0.0916 

56 

-2.16 

0.0352 

Decision*Reasoning 

Excel® 

Knowledge 

Synthesis 

Decision*Reasoning 

Fan  Vis 

Data 

Processing 

Decision*Reasoning 

Fan  Vis 

Information 

Aggregation 

Decision*Reasoning 

Fan  Vis 

Knowledge 

Synthesis 

Type  3  Tests  of  Fixed  Effects 

Effect 

Num  DF 

Den  DF 

F  Value 

Pr>F 

Proficiency  Time 

1 

56 

21.81 

<.0001 

Decision  Support  Tool 

1 

29 

12.17 

0.0016 

Reasoning  Difficulty  Level 

2 

58 

217.71 

<.0001 

Decision*Reasoning 

2 

56 

9.18 

0.0004 
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Tests  of  Between-Subjects  Effects 


Dependent  Variable:  Time 


Source 

Type  III  Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

Corrected  Model 

6378001 ,438a 

5 

1275600.288 

54.972 

.000 

Intercept 

22153966.661 

1 

22153966.661 

954.730 

.000 

Reasoning_Difficulty_Level 

6353097.075 

2 

3176548.538 

136.894 

.000 

Test_Number 

18519.590 

1 

18519.590 

.798 

.373 

Reasoning_Difficulty_Level 
*  TestJMumber 

6384.901 

2 

3192.451 

.138 

.872 

Error 

3991164.166 

172 

23204.443 

Total 

32451639.362 

178 

Corrected  Total 

10369165.604 

177 

a.  R"  =  0.615  (Adjusted  R"  =  0.604) 
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